[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:18 dist-test-slave-1vw0 krb5kdc[23166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:18 dist-test-slave-1vw0 krb5kdc[23166](info): set up 2 sockets
Sep 28 15:50:18 dist-test-slave-1vw0 krb5kdc[23166](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 15:50:19.878798 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:50:20 dist-test-slave-1vw0 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:50:20.069491 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:39129
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:39129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:20.408295 23179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:20.408705 23179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:20.408938 23179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:20.416466 23179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:20.416644 23179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:20.416781 23179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:20.416890 23179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:20.416960 23179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:20.427727 23179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:39129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:39129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:20.429346 23179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:20.430757 23179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:20.444165 23187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:20.444149 23185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:20.444144 23184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:20.445118 23179 server_base.cc:1034] running on GCE node
I20260928 15:50:20.445998 23179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:20.447543 23179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:20.448877 23179 hybrid_clock.cc:648] HybridClock initialized: now 1790610620448825 us; error 41 us; skew 500 ppm
I20260928 15:50:20.449276 23179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:20 dist-test-slave-1vw0 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610620, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:20.452840 23179 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:50:20.454417 23179 webserver.cc:460] Webserver started at http://127.22.157.254:36273/ using document root <none> and password file <none>
I20260928 15:50:20.455166 23179 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:20.455272 23179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:20.455533 23179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:20.459041 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "d202df9965c545db8ff5c828f5ee664f"
format_stamp: "Formatted at 2026-09-28 15:50:20 on dist-test-slave-1vw0"
server_key: "696258d82973d5592422030a52e02303"
server_key_iv: "ff95608b94f888f51ab30cf4837760db"
server_key_version: "encryptionkey@0"
I20260928 15:50:20.460120 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "d202df9965c545db8ff5c828f5ee664f"
format_stamp: "Formatted at 2026-09-28 15:50:20 on dist-test-slave-1vw0"
server_key: "696258d82973d5592422030a52e02303"
server_key_iv: "ff95608b94f888f51ab30cf4837760db"
server_key_version: "encryptionkey@0"
I20260928 15:50:20.466287 23179 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:50:20.470772 23194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:20.472402 23179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:50:20.472723 23179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "d202df9965c545db8ff5c828f5ee664f"
format_stamp: "Formatted at 2026-09-28 15:50:20 on dist-test-slave-1vw0"
server_key: "696258d82973d5592422030a52e02303"
server_key_iv: "ff95608b94f888f51ab30cf4837760db"
server_key_version: "encryptionkey@0"
I20260928 15:50:20.473001 23179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:20.491096 23179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:20.496037 23179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:20.496388 23179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:20.516486 23179 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:39129
I20260928 15:50:20.516520 23245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:39129 every 8 connection(s)
I20260928 15:50:20.518046 23179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:20.522018 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:20.529424 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 23179
I20260928 15:50:20.529779 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:50:20.531870 23159 external_mini_cluster.cc:1428] Setting key 434872f20359ff730e08292078ca0929
I20260928 15:50:20.533221 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f: Bootstrap starting.
I20260928 15:50:20.537072 23246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:20.538282 23246 log.cc:824] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:20.541762 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f: No bootstrap required, opened a new log
I20260928 15:50:20.551795 23246 raft_consensus.cc:348] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d202df9965c545db8ff5c828f5ee664f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39129 } }
I20260928 15:50:20.552266 23246 raft_consensus.cc:374] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:20.552382 23246 raft_consensus.cc:729] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d202df9965c545db8ff5c828f5ee664f, State: Initialized, Role: FOLLOWER
I20260928 15:50:20.553036 23246 consensus_queue.cc:260] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d202df9965c545db8ff5c828f5ee664f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39129 } }
I20260928 15:50:20.553287 23246 raft_consensus.cc:388] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:20.553436 23246 raft_consensus.cc:482] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:20.553611 23246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:20.556991 23246 raft_consensus.cc:504] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d202df9965c545db8ff5c828f5ee664f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39129 } }
I20260928 15:50:20.557538 23246 leader_election.cc:302] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d202df9965c545db8ff5c828f5ee664f; no voters:
I20260928 15:50:20.558231 23246 leader_election.cc:288] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:20.558339 23251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:20.559546 23251 raft_consensus.cc:686] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [term 1 LEADER]: Becoming Leader. State: Replica: d202df9965c545db8ff5c828f5ee664f, State: Running, Role: LEADER
I20260928 15:50:20.560171 23251 consensus_queue.cc:237] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d202df9965c545db8ff5c828f5ee664f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39129 } }
I20260928 15:50:20.560915 23246 sys_catalog.cc:564] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:20.563509 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d202df9965c545db8ff5c828f5ee664f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d202df9965c545db8ff5c828f5ee664f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39129 } } }
I20260928 15:50:20.564006 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:20.565388 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d202df9965c545db8ff5c828f5ee664f. Latest consensus state: current_term: 1 leader_uuid: "d202df9965c545db8ff5c828f5ee664f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d202df9965c545db8ff5c828f5ee664f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39129 } } }
I20260928 15:50:20.565805 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f [sys.catalog]: This master's current role is: LEADER
Sep 28 15:50:20 dist-test-slave-1vw0 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:20.569056 23256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:20.575682 23256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:20.581754 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:20.536191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.22.157.254:39129)
0928 15:50:20.537315 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:20.537333 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:20.538732 (+ 1399us) server_negotiation.cc:407] Connection header received
0928 15:50:20.549156 (+ 10424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:20.549187 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:20.549799 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:20.550330 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:20.553773 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:20.556162 (+ 2389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:20.558329 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:20.559630 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:20.559886 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:20.568140 (+ 8254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:20.568155 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:20.568177 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:20.568236 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:20.573571 (+ 5335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:20.574877 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:20.574887 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:20.574895 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:20.575144 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:20.575865 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:20.575869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:20.575872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:20.576741 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:20.577743 (+ 1002us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:20.579772 (+ 2029us) server_negotiation.cc:299] Negotiation successful
0928 15:50:20.580295 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":504,"threads_started":1}
I20260928 15:50:20.588196 23256 catalog_manager.cc:1347] Generated new cluster ID: 912dda271998417fae1af1a8360c646d
I20260928 15:50:20.588323 23256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:20.623031 23256 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:20.624600 23256 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:20.632171 23256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d202df9965c545db8ff5c828f5ee664f: Generated new TSK 0
I20260928 15:50:20.633186 23256 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:50:20.681263 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39129
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:21.025892 23274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:21.026197 23274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:21.026360 23274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:21.033974 23274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:21.034119 23274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:21.034312 23274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:50:21.044853 23274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:21.046252 23274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:21.047655 23274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:21.061455 23282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:21.061457 23280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:21.062077 23279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:21.062718 23274 server_base.cc:1034] running on GCE node
I20260928 15:50:21.063269 23274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:21.064253 23274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:21.065454 23274 hybrid_clock.cc:648] HybridClock initialized: now 1790610621065386 us; error 64 us; skew 500 ppm
I20260928 15:50:21.065800 23274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:21 dist-test-slave-1vw0 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610621, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:21.074605 23274 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:50:21.076395 23274 webserver.cc:460] Webserver started at http://127.22.157.193:41335/ using document root <none> and password file <none>
I20260928 15:50:21.077394 23274 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:21.077525 23274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:21.077883 23274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:21.081187 23274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "065a12c67a294728a543ba8c693c305b"
format_stamp: "Formatted at 2026-09-28 15:50:21 on dist-test-slave-1vw0"
server_key: "f22b6d39914dec0f30751a1f4086f426"
server_key_iv: "041ee443799a5ace875d75a153f3dbaa"
server_key_version: "encryptionkey@0"
I20260928 15:50:21.081998 23274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "065a12c67a294728a543ba8c693c305b"
format_stamp: "Formatted at 2026-09-28 15:50:21 on dist-test-slave-1vw0"
server_key: "f22b6d39914dec0f30751a1f4086f426"
server_key_iv: "041ee443799a5ace875d75a153f3dbaa"
server_key_version: "encryptionkey@0"
I20260928 15:50:21.087801 23274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:50:21.091758 23289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:21.093401 23274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:21.093591 23274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "065a12c67a294728a543ba8c693c305b"
format_stamp: "Formatted at 2026-09-28 15:50:21 on dist-test-slave-1vw0"
server_key: "f22b6d39914dec0f30751a1f4086f426"
server_key_iv: "041ee443799a5ace875d75a153f3dbaa"
server_key_version: "encryptionkey@0"
I20260928 15:50:21.093829 23274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:21.111778 23274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:21.116490 23274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:21.116922 23274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:21.118060 23274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:21.119588 23274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:21.119671 23274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:21.119817 23274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:21.119889 23274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:21.157580 23274 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:33617
I20260928 15:50:21.157639 23401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:33617 every 8 connection(s)
I20260928 15:50:21.159031 23274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:50:21.161482 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 23274
I20260928 15:50:21.161733 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:50:21.162346 23159 external_mini_cluster.cc:1428] Setting key d8014713bb67c6251a5f30356aacde0c
Sep 28 15:50:21 dist-test-slave-1vw0 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610621, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
I20260928 15:50:21.183049 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:21.162911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:58171 (local address 127.22.157.254:39129)
0928 15:50:21.163393 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:21.163399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:21.163547 (+ 148us) server_negotiation.cc:407] Connection header received
0928 15:50:21.165968 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:21.165972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:21.166042 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:21.166140 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:21.168473 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:21.169459 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:21.171344 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:21.171979 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:21.172103 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:21.176715 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:21.176763 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:21.176768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:21.176813 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:21.179701 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:21.180498 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:21.180506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:21.180514 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:21.180641 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:21.181162 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:21.181172 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:21.181174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:21.181464 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:21.181624 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:21.182448 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 15:50:21.182681 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":959,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
I20260928 15:50:21.184247 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:21.162509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:39129 (local address 127.22.157.193:58171)
0928 15:50:21.163377 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:21.163430 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:21.165627 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:21.166453 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:21.166469 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:21.167365 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:21.168258 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:21.168288 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:21.169646 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:21.169654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:21.171133 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:21.171153 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:21.172261 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:21.172271 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:21.172667 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:21.173503 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:21.173556 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:21.176442 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:21.179860 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:21.179870 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:21.179900 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:21.180355 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:21.180770 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:21.180778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:21.180782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:21.181027 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:21.181593 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:21.181601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:21.182061 (+ 460us) client_negotiation.cc:769] Sending connection context
0928 15:50:21.182344 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 15:50:21.182928 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":470,"threads_started":1}
I20260928 15:50:21.185895 23402 heartbeater.cc:347] Connected to a master server at 127.22.157.254:39129
I20260928 15:50:21.186270 23402 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:21.187227 23402 heartbeater.cc:511] Master 127.22.157.254:39129 requested a full tablet report, sending...
I20260928 15:50:21.190106 23211 ts_manager.cc:194] Registered new tserver with Master: 065a12c67a294728a543ba8c693c305b (127.22.157.193:33617)
I20260928 15:50:21.191643 23211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:58171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:50:21.208496 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39129
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:21.544456 23408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:21.544843 23408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:21.545017 23408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:21.552603 23408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:21.552784 23408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:21.552979 23408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:50:21.563584 23408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:21.565066 23408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:21.566517 23408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:21.580622 23414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:21.580747 23415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:21.580947 23417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:21.582085 23408 server_base.cc:1034] running on GCE node
I20260928 15:50:21.582752 23408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:21.583784 23408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:21.585016 23408 hybrid_clock.cc:648] HybridClock initialized: now 1790610621584973 us; error 43 us; skew 500 ppm
I20260928 15:50:21.585362 23408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:21 dist-test-slave-1vw0 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610621, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:21.588806 23408 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:50:21.590453 23408 webserver.cc:460] Webserver started at http://127.22.157.194:37537/ using document root <none> and password file <none>
I20260928 15:50:21.591354 23408 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:21.591511 23408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:21.591847 23408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:21.595069 23408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "b8fd88dcc1054e1dabe0bdfe64e07323"
format_stamp: "Formatted at 2026-09-28 15:50:21 on dist-test-slave-1vw0"
server_key: "e896907b8303bae48624e1fc314598b7"
server_key_iv: "631ee195bb3b58ea872eb167613845cd"
server_key_version: "encryptionkey@0"
I20260928 15:50:21.595890 23408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "b8fd88dcc1054e1dabe0bdfe64e07323"
format_stamp: "Formatted at 2026-09-28 15:50:21 on dist-test-slave-1vw0"
server_key: "e896907b8303bae48624e1fc314598b7"
server_key_iv: "631ee195bb3b58ea872eb167613845cd"
server_key_version: "encryptionkey@0"
I20260928 15:50:21.602068 23408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260928 15:50:21.605852 23424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:21.607366 23408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:50:21.607594 23408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "b8fd88dcc1054e1dabe0bdfe64e07323"
format_stamp: "Formatted at 2026-09-28 15:50:21 on dist-test-slave-1vw0"
server_key: "e896907b8303bae48624e1fc314598b7"
server_key_iv: "631ee195bb3b58ea872eb167613845cd"
server_key_version: "encryptionkey@0"
I20260928 15:50:21.607834 23408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:21.630183 23408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:21.634960 23408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:21.635294 23408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:21.636339 23408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:21.637888 23408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:21.638001 23408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:21.638161 23408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:21.638237 23408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:21.674002 23408 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:41375
I20260928 15:50:21.674115 23538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:41375 every 8 connection(s)
I20260928 15:50:21.675364 23408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:50:21.676973 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 23408
I20260928 15:50:21.677177 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:50:21.677603 23159 external_mini_cluster.cc:1428] Setting key c2bcba51a92990ceac0ecbd61b6fb29d
Sep 28 15:50:21 dist-test-slave-1vw0 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610621, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
I20260928 15:50:21.700991 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:21.678368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:51551 (local address 127.22.157.254:39129)
0928 15:50:21.678607 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:21.678613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:21.680078 (+ 1465us) server_negotiation.cc:407] Connection header received
0928 15:50:21.683179 (+ 3101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:21.683183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:21.683254 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:21.683341 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:21.685566 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:21.686298 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:21.689019 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:21.689853 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:21.689985 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:21.694155 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:21.694163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:21.694167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:21.694206 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:21.697427 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:21.698355 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:21.698360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:21.698362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:21.698490 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:21.699044 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:21.699050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:21.699052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:21.699402 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:21.699555 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:21.700338 (+ 783us) server_negotiation.cc:299] Negotiation successful
0928 15:50:21.700573 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 15:50:21.702075 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:21.679025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:39129 (local address 127.22.157.194:51551)
0928 15:50:21.679902 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:21.679958 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:21.682840 (+ 2882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:21.683630 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:21.683644 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:21.684457 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:21.685362 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:21.685389 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:21.686545 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:21.686552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:21.688841 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:21.688862 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:21.690094 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:21.690101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:21.690355 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:21.691064 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:21.691086 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:21.693608 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:21.697581 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:21.697591 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:21.697606 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:21.698234 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:21.698595 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:21.698598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:21.698601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:21.698884 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:21.699544 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:21.699552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:21.699936 (+ 384us) client_negotiation.cc:769] Sending connection context
0928 15:50:21.700236 (+ 300us) client_negotiation.cc:240] Negotiation successful
0928 15:50:21.700881 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":335,"threads_started":1}
I20260928 15:50:21.703768 23539 heartbeater.cc:347] Connected to a master server at 127.22.157.254:39129
I20260928 15:50:21.704114 23539 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:21.704945 23539 heartbeater.cc:511] Master 127.22.157.254:39129 requested a full tablet report, sending...
I20260928 15:50:21.706434 23211 ts_manager.cc:194] Registered new tserver with Master: b8fd88dcc1054e1dabe0bdfe64e07323 (127.22.157.194:41375)
I20260928 15:50:21.707259 23211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:51551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:50:21.722316 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39129
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:34791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:22.059569 23544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:22.059878 23544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:22.059983 23544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:22.067545 23544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:22.067677 23544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:22.067797 23544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:50:22.078244 23544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39129
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:22.079654 23544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:22.081046 23544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:22.095715 23549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:22.095783 23550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:22.095749 23552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:22.096576 23544 server_base.cc:1034] running on GCE node
I20260928 15:50:22.097324 23544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:22.098320 23544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:22.099571 23544 hybrid_clock.cc:648] HybridClock initialized: now 1790610622099504 us; error 47 us; skew 500 ppm
I20260928 15:50:22.099905 23544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610622, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:22.103216 23544 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:50:22.104911 23544 webserver.cc:460] Webserver started at http://127.22.157.195:38193/ using document root <none> and password file <none>
I20260928 15:50:22.105682 23544 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:22.105834 23544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:22.106155 23544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:22.109191 23544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "9c7dd53b65c74f5d8736edaec35cbc4d"
format_stamp: "Formatted at 2026-09-28 15:50:22 on dist-test-slave-1vw0"
server_key: "474f7cd1dccb43da034218ed0e0d2859"
server_key_iv: "25331ee859ac0c4c6a201ac7cafb4493"
server_key_version: "encryptionkey@0"
I20260928 15:50:22.109972 23544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "9c7dd53b65c74f5d8736edaec35cbc4d"
format_stamp: "Formatted at 2026-09-28 15:50:22 on dist-test-slave-1vw0"
server_key: "474f7cd1dccb43da034218ed0e0d2859"
server_key_iv: "25331ee859ac0c4c6a201ac7cafb4493"
server_key_version: "encryptionkey@0"
I20260928 15:50:22.115408 23544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:50:22.119040 23559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:22.120422 23544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:50:22.120620 23544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "9c7dd53b65c74f5d8736edaec35cbc4d"
format_stamp: "Formatted at 2026-09-28 15:50:22 on dist-test-slave-1vw0"
server_key: "474f7cd1dccb43da034218ed0e0d2859"
server_key_iv: "25331ee859ac0c4c6a201ac7cafb4493"
server_key_version: "encryptionkey@0"
I20260928 15:50:22.120872 23544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:22.137979 23544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:22.142009 23544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:22.142302 23544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:22.143288 23544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:22.144743 23544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:22.144847 23544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:22.145006 23544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:22.145087 23544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:22.181394 23544 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:43789
I20260928 15:50:22.181453 23675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:43789 every 8 connection(s)
I20260928 15:50:22.182823 23544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:50:22.190987 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 23544
I20260928 15:50:22.191226 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:50:22.191684 23159 external_mini_cluster.cc:1428] Setting key 6d6556fbf6e169f0296832c724270273
I20260928 15:50:22.194761 23402 heartbeater.cc:503] Master 127.22.157.254:39129 was elected leader, sending a full tablet report...
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610622, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:22.205200 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:22.185516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:35465 (local address 127.22.157.254:39129)
0928 15:50:22.185744 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:22.185751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:22.187082 (+ 1331us) server_negotiation.cc:407] Connection header received
0928 15:50:22.189029 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:22.189034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:22.189143 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:22.189241 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:22.190935 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.192008 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.193914 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.194516 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.194672 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:22.199127 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:22.199134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:22.199138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:22.199174 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:22.201918 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.202757 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.202762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.202764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.202913 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.203418 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.203423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.203425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.203717 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:22.203829 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:22.204649 (+ 820us) server_negotiation.cc:299] Negotiation successful
0928 15:50:22.204931 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:50:22.206298 23678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:22.186165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:39129 (local address 127.22.157.195:35465)
0928 15:50:22.186909 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:22.186953 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:22.188670 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:22.189464 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:22.189472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:22.189999 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:22.190682 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:22.190712 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.192205 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.192213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:22.193745 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:22.193762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.194789 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.194800 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:22.195165 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:22.196083 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:22.196118 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:22.198881 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:22.202097 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:22.202107 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:22.202126 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:22.202611 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:22.203075 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:22.203079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:22.203081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:22.203289 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:22.203823 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:22.203829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:22.204187 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 15:50:22.204455 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 15:50:22.205089 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":296,"threads_started":1}
I20260928 15:50:22.208048 23676 heartbeater.cc:347] Connected to a master server at 127.22.157.254:39129
I20260928 15:50:22.208379 23676 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:22.209204 23676 heartbeater.cc:511] Master 127.22.157.254:39129 requested a full tablet report, sending...
I20260928 15:50:22.210647 23211 ts_manager.cc:194] Registered new tserver with Master: 9c7dd53b65c74f5d8736edaec35cbc4d (127.22.157.195:43789)
I20260928 15:50:22.211679 23211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:35465
I20260928 15:50:22.221230 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610622, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610622, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:50:22.269912 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:22.252896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.22.157.193:33617)
0928 15:50:22.253607 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:22.253617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:22.253649 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:50:22.253838 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:22.253845 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:22.254083 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:22.254265 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:22.255796 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.256961 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.258423 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.259035 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.259180 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:22.263600 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:22.263614 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:22.263630 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:22.263677 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:22.266926 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.267696 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.267704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.267706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.267833 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.268176 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.268179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.268181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.268557 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:22.268895 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:22.269255 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 15:50:22.269537 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":300,"threads_started":1}
W20260928 15:50:22.271265 23317 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55200
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:50:22.330121 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:22.311864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.22.157.193:33617)
0928 15:50:22.312110 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:22.312119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:22.312423 (+ 304us) server_negotiation.cc:407] Connection header received
0928 15:50:22.312662 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:22.312670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:22.312797 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:22.312945 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:22.314311 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.315623 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.316992 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.317847 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.318021 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:22.322553 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:22.322562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:22.322569 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:22.322626 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:22.326871 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.327657 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.327666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.327671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.327859 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.328304 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.328313 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.328318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.328717 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:22.329044 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:22.329365 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 15:50:22.329653 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 15:50:22.332546 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 23274
I20260928 15:50:22.349668 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:22.333807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.22.157.193:33617)
0928 15:50:22.334057 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:22.334064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:22.334230 (+ 166us) server_negotiation.cc:407] Connection header received
0928 15:50:22.334368 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:22.334375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:22.334471 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:22.334581 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:22.335798 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.337161 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.338537 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.339416 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.339571 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:22.342128 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:22.342137 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:22.342145 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:22.342198 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:22.346336 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.347185 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.347193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.347199 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.347398 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.347803 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.347811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.347817 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.348277 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:22.348589 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:22.348878 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 15:50:22.349165 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 15:50:22.350572 23397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:22.511637 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 23274
I20260928 15:50:22.532588 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 23408
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:50:22.553412 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:22.533841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.22.157.194:41375)
0928 15:50:22.535006 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:22.535020 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:22.535061 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:50:22.535156 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:22.535170 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:22.535513 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:22.535721 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:22.537016 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.539017 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.540338 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.541529 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.541708 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:22.546716 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:22.546731 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:22.546747 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:22.546795 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:22.550022 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.551107 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.551115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.551118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.551246 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.551731 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.551735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.551736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.552099 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:22.552337 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:22.552788 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 15:50:22.553006 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":740,"threads_started":1}
I20260928 15:50:22.555207 23534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:22.641944 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 23408
I20260928 15:50:22.659166 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 23544
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:50:22.676936 23692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:22.660506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.22.157.195:43789)
0928 15:50:22.661160 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:22.661168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:22.661198 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:50:22.661258 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:22.661264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:22.661526 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:22.661673 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:22.663042 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.664072 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.665635 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:22.666199 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:22.666299 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:22.670813 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:22.670829 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:22.670845 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:22.670911 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:22.673847 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.674673 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.674689 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.674691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.674928 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:22.675368 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:22.675372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:22.675374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:22.675765 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:22.676030 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:22.676310 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 15:50:22.676537 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":360,"threads_started":1}
I20260928 15:50:22.678359 23671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:22.763266 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 23544
I20260928 15:50:22.779700 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 23179
I20260928 15:50:22.780949 23241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:22.885684 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 23179
2026-09-28T15:50:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4114 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23697](info): set up 2 sockets
Sep 28 15:50:22 dist-test-slave-1vw0 krb5kdc[23697](info): commencing operation
krb5kdc: starting...
W20260928 15:50:23.989187 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:50:24 dist-test-slave-1vw0 krb5kdc[23697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:50:24.189291 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:46227
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:24.516058 23711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:24.516359 23711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:24.516462 23711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:24.523854 23711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:24.523967 23711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:24.524014 23711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:24.524065 23711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:24.524111 23711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:24.534318 23711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:46227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:24.535641 23711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:24.536937 23711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:24.549818 23717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:24.549856 23719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:24.549786 23716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:24.551149 23711 server_base.cc:1034] running on GCE node
I20260928 15:50:24.551980 23711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:24.553516 23711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:24.554725 23711 hybrid_clock.cc:648] HybridClock initialized: now 1790610624554680 us; error 33 us; skew 500 ppm
I20260928 15:50:24.555112 23711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:24 dist-test-slave-1vw0 krb5kdc[23697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610624, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:24.558257 23711 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:50:24.559763 23711 webserver.cc:460] Webserver started at http://127.22.157.254:37933/ using document root <none> and password file <none>
I20260928 15:50:24.560501 23711 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:24.560649 23711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:24.560994 23711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:24.564023 23711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "bd80462faaaa40a8b7087b933264d8f9"
format_stamp: "Formatted at 2026-09-28 15:50:24 on dist-test-slave-1vw0"
server_key: "bdd4a5ee5c33e2ed48ecb464c1a0984d"
server_key_iv: "a85b55a18dad5019444b0cdcb974435b"
server_key_version: "encryptionkey@0"
I20260928 15:50:24.564869 23711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "bd80462faaaa40a8b7087b933264d8f9"
format_stamp: "Formatted at 2026-09-28 15:50:24 on dist-test-slave-1vw0"
server_key: "bdd4a5ee5c33e2ed48ecb464c1a0984d"
server_key_iv: "a85b55a18dad5019444b0cdcb974435b"
server_key_version: "encryptionkey@0"
I20260928 15:50:24.570225 23711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 15:50:24.573855 23726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:24.575291 23711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:50:24.575477 23711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "bd80462faaaa40a8b7087b933264d8f9"
format_stamp: "Formatted at 2026-09-28 15:50:24 on dist-test-slave-1vw0"
server_key: "bdd4a5ee5c33e2ed48ecb464c1a0984d"
server_key_iv: "a85b55a18dad5019444b0cdcb974435b"
server_key_version: "encryptionkey@0"
I20260928 15:50:24.575680 23711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:24.610067 23711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:24.614533 23711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:24.614845 23711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:24.634270 23711 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:46227
I20260928 15:50:24.634300 23779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:46227 every 8 connection(s)
I20260928 15:50:24.635778 23711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:24.639851 23780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:24.643456 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 23711
I20260928 15:50:24.643743 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:50:24.644680 23159 external_mini_cluster.cc:1428] Setting key 97fe8fc47619c8c762c69e4eeb8ab267
I20260928 15:50:24.653363 23780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9: Bootstrap starting.
I20260928 15:50:24.657109 23780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:24.658170 23780 log.cc:824] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:50:24 dist-test-slave-1vw0 krb5kdc[23697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:24.661752 23780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9: No bootstrap required, opened a new log
I20260928 15:50:24.669010 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:24.647149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.22.157.254:46227)
0928 15:50:24.647978 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:24.647991 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:24.648039 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:50:24.649849 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:24.649872 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:24.650304 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:24.650721 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:24.652196 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:24.653637 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:24.655108 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:24.655800 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:24.656009 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:24.660573 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:24.660588 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:24.660607 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:24.660657 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:24.663997 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:24.664923 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:24.664930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:24.664937 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:24.665128 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:24.665545 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:24.665549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:24.665551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:24.666115 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:24.666336 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:24.666938 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 15:50:24.667363 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":407,"threads_started":1}
I20260928 15:50:24.671995 23780 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd80462faaaa40a8b7087b933264d8f9" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46227 } }
I20260928 15:50:24.672412 23780 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:24.672533 23780 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd80462faaaa40a8b7087b933264d8f9, State: Initialized, Role: FOLLOWER
I20260928 15:50:24.673156 23780 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd80462faaaa40a8b7087b933264d8f9" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46227 } }
I20260928 15:50:24.673431 23780 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:24.673597 23780 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:24.673774 23780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:24.676020 23780 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd80462faaaa40a8b7087b933264d8f9" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46227 } }
I20260928 15:50:24.676568 23780 leader_election.cc:302] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd80462faaaa40a8b7087b933264d8f9; no voters:
I20260928 15:50:24.677209 23780 leader_election.cc:288] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:24.677337 23786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:24.678081 23786 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [term 1 LEADER]: Becoming Leader. State: Replica: bd80462faaaa40a8b7087b933264d8f9, State: Running, Role: LEADER
I20260928 15:50:24.678658 23786 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd80462faaaa40a8b7087b933264d8f9" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46227 } }
I20260928 15:50:24.679576 23780 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:24.682175 23787 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd80462faaaa40a8b7087b933264d8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd80462faaaa40a8b7087b933264d8f9" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46227 } } }
I20260928 15:50:24.682155 23788 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd80462faaaa40a8b7087b933264d8f9. Latest consensus state: current_term: 1 leader_uuid: "bd80462faaaa40a8b7087b933264d8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd80462faaaa40a8b7087b933264d8f9" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46227 } } }
I20260928 15:50:24.682641 23787 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:24.682669 23788 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:24.692504 23791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:24.696957 23791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:24.707162 23791 catalog_manager.cc:1347] Generated new cluster ID: 70416aad8e764a6d8a01b172429355bd
I20260928 15:50:24.707288 23791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:24.737107 23791 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:24.738519 23791 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:24.745481 23791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd80462faaaa40a8b7087b933264d8f9: Generated new TSK 0
I20260928 15:50:24.746385 23791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:50:24.800992 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46227
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:25.134263 23807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:25.134574 23807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:25.134730 23807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:25.142321 23807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:25.142441 23807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:25.142637 23807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:50:25.152899 23807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:25.154321 23807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:25.155689 23807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:25.169714 23815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:25.169770 23813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:25.169978 23812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:25.170527 23807 server_base.cc:1034] running on GCE node
I20260928 15:50:25.171294 23807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:25.172314 23807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:25.173530 23807 hybrid_clock.cc:648] HybridClock initialized: now 1790610625173447 us; error 72 us; skew 500 ppm
I20260928 15:50:25.173822 23807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:25 dist-test-slave-1vw0 krb5kdc[23697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610625, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:25.177266 23807 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:50:25.178875 23807 webserver.cc:460] Webserver started at http://127.22.157.193:43091/ using document root <none> and password file <none>
I20260928 15:50:25.179764 23807 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:25.179908 23807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:25.180233 23807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:25.183486 23807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "ff3c26fdd0334804bc1c6051b2fea291"
format_stamp: "Formatted at 2026-09-28 15:50:25 on dist-test-slave-1vw0"
server_key: "e1a593d967fa688809a0c0e94b1e5f54"
server_key_iv: "165f3d64e70132e7278190cc55ea695f"
server_key_version: "encryptionkey@0"
I20260928 15:50:25.184307 23807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "ff3c26fdd0334804bc1c6051b2fea291"
format_stamp: "Formatted at 2026-09-28 15:50:25 on dist-test-slave-1vw0"
server_key: "e1a593d967fa688809a0c0e94b1e5f54"
server_key_iv: "165f3d64e70132e7278190cc55ea695f"
server_key_version: "encryptionkey@0"
I20260928 15:50:25.189910 23807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:50:25.193635 23822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:25.195144 23807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:50:25.195326 23807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "ff3c26fdd0334804bc1c6051b2fea291"
format_stamp: "Formatted at 2026-09-28 15:50:25 on dist-test-slave-1vw0"
server_key: "e1a593d967fa688809a0c0e94b1e5f54"
server_key_iv: "165f3d64e70132e7278190cc55ea695f"
server_key_version: "encryptionkey@0"
I20260928 15:50:25.195520 23807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:25.209735 23807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:25.214076 23807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:25.214389 23807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:25.215466 23807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:25.216895 23807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:25.216966 23807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:25.217092 23807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:25.217154 23807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:25.253122 23807 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:38123
I20260928 15:50:25.253140 23937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:38123 every 8 connection(s)
I20260928 15:50:25.254436 23807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:50:25.260068 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 23807
I20260928 15:50:25.260298 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:50:25.260977 23159 external_mini_cluster.cc:1428] Setting key cb8fb9f34dd042a2238aeac36134757e
Sep 28 15:50:25 dist-test-slave-1vw0 krb5kdc[23697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610625, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:25.277133 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:25.257157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:55059 (local address 127.22.157.254:46227)
0928 15:50:25.257765 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:25.257770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:25.258645 (+ 875us) server_negotiation.cc:407] Connection header received
0928 15:50:25.261138 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:25.261144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:25.261239 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:25.261365 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:25.263187 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:25.263917 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:25.265605 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:25.266004 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:25.266138 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:25.270108 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:25.270115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:25.270119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:25.270161 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:25.273307 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:25.274414 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:25.274418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:25.274421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:25.274558 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:25.275114 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:25.275118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:25.275120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:25.275402 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:25.275544 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:25.276620 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0928 15:50:25.276869 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":365,"threads_started":1}
I20260928 15:50:25.278754 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:25.257650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46227 (local address 127.22.157.193:55059)
0928 15:50:25.258448 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:25.258497 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:25.260841 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:25.261625 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:25.261634 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:25.262307 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:25.263020 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:25.263039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:25.264083 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:25.264088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:25.265441 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:25.265453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:25.266243 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:25.266250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:25.266486 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:25.267200 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:25.267223 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:25.269832 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:25.273489 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:25.273501 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:25.273524 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:25.274269 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:25.274680 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:25.274688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:25.274695 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:25.274981 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:25.275541 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:25.275552 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:25.276154 (+ 602us) client_negotiation.cc:769] Sending connection context
0928 15:50:25.276515 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 15:50:25.277206 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":294,"threads_started":1}
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
I20260928 15:50:25.280582 23938 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46227
I20260928 15:50:25.280968 23938 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:25.281921 23938 heartbeater.cc:511] Master 127.22.157.254:46227 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
I20260928 15:50:25.284439 23743 ts_manager.cc:194] Registered new tserver with Master: ff3c26fdd0334804bc1c6051b2fea291 (127.22.157.193:38123)
I20260928 15:50:25.286177 23743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:55059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:50:25.309423 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46227
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:25.638000 23944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:25.638432 23944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:25.638588 23944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:25.650410 23944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:25.650552 23944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:25.650739 23944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:50:25.663019 23944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:25.664387 23944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:25.665875 23944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:25.679359 23949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:25.679330 23950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:25.679383 23952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:25.680177 23944 server_base.cc:1034] running on GCE node
I20260928 15:50:25.680894 23944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:25.681859 23944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:25.683086 23944 hybrid_clock.cc:648] HybridClock initialized: now 1790610625683027 us; error 45 us; skew 500 ppm
I20260928 15:50:25.683414 23944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:25 dist-test-slave-1vw0 krb5kdc[23697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610625, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:25.686793 23944 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:50:25.688412 23944 webserver.cc:460] Webserver started at http://127.22.157.194:35639/ using document root <none> and password file <none>
I20260928 15:50:25.689292 23944 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:25.689428 23944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:25.689702 23944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:25.692806 23944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "d9a2464d933940528b5ed4a0d109527f"
format_stamp: "Formatted at 2026-09-28 15:50:25 on dist-test-slave-1vw0"
server_key: "37cef9b4a230133ece8c0f5f908c71b1"
server_key_iv: "93c9bbdb08b539f204100d2b1851e199"
server_key_version: "encryptionkey@0"
I20260928 15:50:25.693717 23944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "d9a2464d933940528b5ed4a0d109527f"
format_stamp: "Formatted at 2026-09-28 15:50:25 on dist-test-slave-1vw0"
server_key: "37cef9b4a230133ece8c0f5f908c71b1"
server_key_iv: "93c9bbdb08b539f204100d2b1851e199"
server_key_version: "encryptionkey@0"
I20260928 15:50:25.699368 23944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:50:25.703799 23959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:25.705677 23944 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 15:50:25.705909 23944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "d9a2464d933940528b5ed4a0d109527f"
format_stamp: "Formatted at 2026-09-28 15:50:25 on dist-test-slave-1vw0"
server_key: "37cef9b4a230133ece8c0f5f908c71b1"
server_key_iv: "93c9bbdb08b539f204100d2b1851e199"
server_key_version: "encryptionkey@0"
I20260928 15:50:25.706192 23944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:25.723585 23944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:25.728123 23944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:25.728410 23944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:25.729472 23944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:25.730857 23944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:25.730947 23944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:25.731038 23944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:25.731086 23944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:25.764082 23944 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:44773
I20260928 15:50:25.764139 24073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:44773 every 8 connection(s)
I20260928 15:50:25.765563 23944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:50:25.767868 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 23944
I20260928 15:50:25.768080 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:50:25.768636 23159 external_mini_cluster.cc:1428] Setting key 1de4d39e881a3914e4a62575baa65b9b
Sep 28 15:50:25 dist-test-slave-1vw0 krb5kdc[23697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610625, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
I20260928 15:50:25.796557 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:25.768320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:48355 (local address 127.22.157.254:46227)
0928 15:50:25.768553 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:25.768560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:25.770812 (+ 2252us) server_negotiation.cc:407] Connection header received
0928 15:50:25.774468 (+ 3656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:25.774477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:25.774621 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:25.774745 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:25.777802 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:25.778718 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:25.781948 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:25.782808 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:25.782961 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:25.788434 (+ 5473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:25.788442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:25.788447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:25.788501 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:25.792109 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:25.793308 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:25.793315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:25.793320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:25.793503 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:25.794172 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:25.794176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:25.794179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:25.794531 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:25.794655 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:25.795807 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0928 15:50:25.796080 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:50:25.797844 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:25.768969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46227 (local address 127.22.157.194:48355)
0928 15:50:25.770660 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:25.770709 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:25.774028 (+ 3319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:25.775116 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:25.775132 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:25.776293 (+ 1161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:25.777505 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:25.777568 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:25.778903 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:25.778909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:25.781735 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:25.781756 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:25.783318 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:25.783327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:25.783699 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:25.784647 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:25.784678 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:25.788097 (+ 3419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:25.792336 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:25.792349 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:25.792372 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:25.793139 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:25.793695 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:25.793702 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:25.793708 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:25.793996 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:25.794663 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:25.794675 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:25.795259 (+ 584us) client_negotiation.cc:769] Sending connection context
0928 15:50:25.795623 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 15:50:25.796257 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1381,"thread_start_us":322,"threads_started":1}
I20260928 15:50:25.799623 24074 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46227
I20260928 15:50:25.799969 24074 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:25.801024 24074 heartbeater.cc:511] Master 127.22.157.254:46227 requested a full tablet report, sending...
I20260928 15:50:25.803501 23743 ts_manager.cc:194] Registered new tserver with Master: d9a2464d933940528b5ed4a0d109527f (127.22.157.194:44773)
I20260928 15:50:25.804848 23743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:48355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:50:25.828015 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46227
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:26.152814 24079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:26.153154 24079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:26.153323 24079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:26.160782 24079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:26.160972 24079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:26.161159 24079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:50:26.171110 24079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46227
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:26.172482 24079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:26.173934 24079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:26.187515 24085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:26.187566 24087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:26.187449 24084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:26.188436 24079 server_base.cc:1034] running on GCE node
I20260928 15:50:26.189143 24079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:26.190145 24079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:26.191378 24079 hybrid_clock.cc:648] HybridClock initialized: now 1790610626191332 us; error 42 us; skew 500 ppm
I20260928 15:50:26.191706 24079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:26 dist-test-slave-1vw0 krb5kdc[23697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610626, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:26.200877 24079 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:50:26.202749 24079 webserver.cc:460] Webserver started at http://127.22.157.195:37391/ using document root <none> and password file <none>
I20260928 15:50:26.203542 24079 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:26.203701 24079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:26.204042 24079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:26.207242 24079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "07f7f81376474626bc8745c9970f066d"
format_stamp: "Formatted at 2026-09-28 15:50:26 on dist-test-slave-1vw0"
server_key: "9dcf4ba240ca1367654265e28aedbbde"
server_key_iv: "4f1912a156336c802ab82d6eccf261b8"
server_key_version: "encryptionkey@0"
I20260928 15:50:26.208102 24079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "07f7f81376474626bc8745c9970f066d"
format_stamp: "Formatted at 2026-09-28 15:50:26 on dist-test-slave-1vw0"
server_key: "9dcf4ba240ca1367654265e28aedbbde"
server_key_iv: "4f1912a156336c802ab82d6eccf261b8"
server_key_version: "encryptionkey@0"
I20260928 15:50:26.214061 24079 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 15:50:26.217783 24094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:26.219206 24079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:50:26.219398 24079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "07f7f81376474626bc8745c9970f066d"
format_stamp: "Formatted at 2026-09-28 15:50:26 on dist-test-slave-1vw0"
server_key: "9dcf4ba240ca1367654265e28aedbbde"
server_key_iv: "4f1912a156336c802ab82d6eccf261b8"
server_key_version: "encryptionkey@0"
I20260928 15:50:26.219632 24079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:26.249398 24079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:26.254009 24079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:26.254328 24079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:26.255331 24079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:26.256742 24079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:26.256876 24079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:26.257033 24079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:26.257115 24079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:26.290040 23938 heartbeater.cc:503] Master 127.22.157.254:46227 was elected leader, sending a full tablet report...
I20260928 15:50:26.292220 24079 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:35291
I20260928 15:50:26.292258 24207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:35291 every 8 connection(s)
I20260928 15:50:26.293605 24079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:50:26.296834 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 24079
I20260928 15:50:26.297055 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:50:26.297677 23159 external_mini_cluster.cc:1428] Setting key b7e561886ae0394d4f684fc8a0c791f4
Sep 28 15:50:26 dist-test-slave-1vw0 krb5kdc[23697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610626, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:26.315153 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.296199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:59303 (local address 127.22.157.254:46227)
0928 15:50:26.296402 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.296407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.297978 (+ 1571us) server_negotiation.cc:407] Connection header received
0928 15:50:26.300396 (+ 2418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.300401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.300524 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.300649 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:26.302333 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.303110 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.304699 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.305181 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.305323 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.308785 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:26.308790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:26.308793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:26.308826 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:26.311642 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.312758 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.312766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.312769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.312938 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.313472 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.313478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.313481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.313880 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:26.313984 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.314725 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.314886 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:50:26.316162 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.296935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46227 (local address 127.22.157.195:59303)
0928 15:50:26.297797 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:26.297859 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:26.300081 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:26.300940 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:26.300955 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:26.301553 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:26.302159 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.302178 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.303273 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.303279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.304545 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.304557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.305418 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.305427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.305687 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.306318 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:26.306340 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:26.308540 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:26.311806 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:26.311813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:26.311827 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:26.312608 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:26.313077 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:26.313082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:26.313085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:26.313364 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:26.314012 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:26.314018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:26.314357 (+ 339us) client_negotiation.cc:769] Sending connection context
0928 15:50:26.314644 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 15:50:26.315039 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":330,"threads_started":1}
I20260928 15:50:26.317770 24208 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46227
I20260928 15:50:26.318085 24208 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:26.318753 24208 heartbeater.cc:511] Master 127.22.157.254:46227 requested a full tablet report, sending...
I20260928 15:50:26.320168 23743 ts_manager.cc:194] Registered new tserver with Master: 07f7f81376474626bc8745c9970f066d (127.22.157.195:35291)
I20260928 15:50:26.321019 23743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:59303
I20260928 15:50:26.326100 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:50:26.348225 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.332552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.22.157.254:46227)
0928 15:50:26.332791 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.332801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.333253 (+ 452us) server_negotiation.cc:407] Connection header received
0928 15:50:26.333389 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.333392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.333456 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.333541 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:26.335075 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.335937 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.337381 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.338015 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.338158 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.341416 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:26.341429 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:26.341433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:26.341491 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:26.345005 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.345948 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.345954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.345958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.346161 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.346613 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.346616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.346618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.347128 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:26.347430 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.347810 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.347988 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:50:26.355978 23743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:50:26.363188 23743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:50:26.401577 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.377626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.22.157.194:44773)
0928 15:50:26.389264 (+ 11638us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.389278 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.389321 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:50:26.389470 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.389482 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.389859 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.392076 (+ 2217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:26.392373 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.394506 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.398463 (+ 3957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.400159 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.400392 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.400581 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.401024 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.401198 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11458,"thread_start_us":11310,"threads_started":1}
I20260928 15:50:26.401628 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.376880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:44773 (local address 127.0.0.1:53632)
0928 15:50:26.384574 (+ 7694us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:26.384636 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:26.385629 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:26.391136 (+ 5507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:26.391149 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:26.391212 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:26.391755 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.391774 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.395606 (+ 3832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.395611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.398259 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.398277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.400475 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.400485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.400752 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.400809 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:50:26.400955 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 15:50:26.401177 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7360,"mutex_wait_us":3,"thread_start_us":18257,"threads_started":1}
I20260928 15:50:26.404567 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.378137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.22.157.195:35291)
0928 15:50:26.394763 (+ 16626us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.394773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.395265 (+ 492us) server_negotiation.cc:407] Connection header received
0928 15:50:26.395622 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.395631 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.395925 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.396080 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:26.397336 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.398858 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.401560 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.403055 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.403229 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.403396 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.404051 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.404233 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16405,"thread_start_us":16647,"threads_started":1}
I20260928 15:50:26.404847 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.378117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.22.157.193:38123)
0928 15:50:26.390688 (+ 12571us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.390701 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.395639 (+ 4938us) server_negotiation.cc:407] Connection header received
0928 15:50:26.395774 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.395784 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.396048 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.396275 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:26.397255 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.398836 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.401946 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.403399 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.403613 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.403780 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.404301 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.404487 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12328,"thread_start_us":12593,"threads_started":1}
I20260928 15:50:26.405073 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.377823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:38123 (local address 127.0.0.1:55750)
0928 15:50:26.395469 (+ 17646us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:26.395487 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:26.395618 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:26.396274 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:26.396278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:26.396304 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:26.397011 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.397026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.398978 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.398985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.401102 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.401115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.403666 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.403674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.403843 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.403882 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:50:26.403968 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:50:26.404105 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17506,"mutex_wait_us":142,"thread_start_us":16967,"threads_started":1}
I20260928 15:50:26.405275 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.377151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:35291 (local address 127.0.0.1:35654)
0928 15:50:26.395125 (+ 17974us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:26.395153 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:26.395394 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:26.396439 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:26.396445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:26.396489 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:26.397101 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.397111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.399049 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.399054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.401355 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.401372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.403639 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.403646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.403835 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.403878 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:50:26.403979 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:50:26.404114 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17785,"mutex_wait_us":6,"spinlock_wait_cycles":3968,"thread_start_us":1370,"threads_started":1}
I20260928 15:50:26.406665 24007 tablet_service.cc:1467] Processing CreateTablet for tablet 7c290f0910004b818ffd17de0228d0da (DEFAULT_TABLE table=test-table [id=2583cb309715418d9adf0e7b63d10310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:26.409014 24007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c290f0910004b818ffd17de0228d0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:26.409097 23870 tablet_service.cc:1467] Processing CreateTablet for tablet 7c290f0910004b818ffd17de0228d0da (DEFAULT_TABLE table=test-table [id=2583cb309715418d9adf0e7b63d10310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:26.409273 24142 tablet_service.cc:1467] Processing CreateTablet for tablet 7c290f0910004b818ffd17de0228d0da (DEFAULT_TABLE table=test-table [id=2583cb309715418d9adf0e7b63d10310]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:26.410547 23870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c290f0910004b818ffd17de0228d0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:26.410734 24142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c290f0910004b818ffd17de0228d0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:26.421954 24227 tablet_bootstrap.cc:492] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291: Bootstrap starting.
I20260928 15:50:26.425326 24228 tablet_bootstrap.cc:492] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f: Bootstrap starting.
I20260928 15:50:26.426335 24227 tablet_bootstrap.cc:654] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:26.426563 24229 tablet_bootstrap.cc:492] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d: Bootstrap starting.
I20260928 15:50:26.427862 24227 log.cc:824] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:26.430770 24229 tablet_bootstrap.cc:654] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:26.430902 24227 tablet_bootstrap.cc:492] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291: No bootstrap required, opened a new log
I20260928 15:50:26.431172 24227 ts_tablet_manager.cc:1397] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260928 15:50:26.431999 24228 tablet_bootstrap.cc:654] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:26.432156 24229 log.cc:824] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:26.433241 24228 log.cc:824] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:26.435467 24229 tablet_bootstrap.cc:492] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d: No bootstrap required, opened a new log
I20260928 15:50:26.435810 24229 ts_tablet_manager.cc:1397] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260928 15:50:26.436787 24228 tablet_bootstrap.cc:492] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f: No bootstrap required, opened a new log
I20260928 15:50:26.437063 24228 ts_tablet_manager.cc:1397] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260928 15:50:26.439903 24227 raft_consensus.cc:348] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } }
I20260928 15:50:26.440284 24227 raft_consensus.cc:374] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:26.440375 24227 raft_consensus.cc:729] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3c26fdd0334804bc1c6051b2fea291, State: Initialized, Role: FOLLOWER
I20260928 15:50:26.440989 24227 consensus_queue.cc:260] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } }
I20260928 15:50:26.442895 24227 ts_tablet_manager.cc:1428] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260928 15:50:26.447439 24229 raft_consensus.cc:348] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } }
I20260928 15:50:26.447989 24229 raft_consensus.cc:374] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:26.448096 24229 raft_consensus.cc:729] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f7f81376474626bc8745c9970f066d, State: Initialized, Role: FOLLOWER
I20260928 15:50:26.447907 24228 raft_consensus.cc:348] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } }
I20260928 15:50:26.448393 24228 raft_consensus.cc:374] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:26.448515 24228 raft_consensus.cc:729] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a2464d933940528b5ed4a0d109527f, State: Initialized, Role: FOLLOWER
I20260928 15:50:26.448967 24229 consensus_queue.cc:260] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } }
I20260928 15:50:26.449220 24228 consensus_queue.cc:260] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } }
I20260928 15:50:26.450208 24208 heartbeater.cc:503] Master 127.22.157.254:46227 was elected leader, sending a full tablet report...
I20260928 15:50:26.451256 24229 ts_tablet_manager.cc:1428] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 15:50:26.453387 24074 heartbeater.cc:503] Master 127.22.157.254:46227 was elected leader, sending a full tablet report...
I20260928 15:50:26.454213 24228 ts_tablet_manager.cc:1428] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260928 15:50:26.463430 24235 raft_consensus.cc:482] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:26.463742 24235 raft_consensus.cc:504] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } }
I20260928 15:50:26.465538 24235 leader_election.cc:288] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07f7f81376474626bc8745c9970f066d (127.22.157.195:35291), ff3c26fdd0334804bc1c6051b2fea291 (127.22.157.193:38123)
I20260928 15:50:26.472981 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.466146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:50641 (local address 127.22.157.195:35291)
0928 15:50:26.466375 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.466382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.466845 (+ 463us) server_negotiation.cc:407] Connection header received
0928 15:50:26.467060 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.467067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.467176 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.467272 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:26.468127 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.469221 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.471181 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.471948 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.472140 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.472225 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.472552 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.472689 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:50:26.473012 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.466086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:34139 (local address 127.22.157.193:38123)
0928 15:50:26.466451 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.466457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.467035 (+ 578us) server_negotiation.cc:407] Connection header received
0928 15:50:26.467210 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.467218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.467294 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.467380 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:26.468174 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.469236 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.471172 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.471950 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.472145 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.472242 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.472572 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.472706 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260928 15:50:26.473160 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.465937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:38123 (local address 127.22.157.194:34139)
0928 15:50:26.466923 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:26.466942 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:26.467066 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:26.467384 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:26.467387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:26.467404 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:26.467944 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.467952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.469384 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.469390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.471008 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.471021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.472216 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.472222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.472408 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.472431 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:50:26.472547 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 15:50:26.472697 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":7,"thread_start_us":625,"threads_started":1}
I20260928 15:50:26.473595 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.465852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:35291 (local address 127.22.157.194:50641)
0928 15:50:26.466692 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:26.466714 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:26.466880 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:26.467354 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:26.467360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:26.467388 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:26.467913 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.467933 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.469344 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.469351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.471024 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:26.471036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.472189 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.472200 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:26.472384 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.472410 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:50:26.472519 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:50:26.472683 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":116,"thread_start_us":443,"threads_started":1}
I20260928 15:50:26.474418 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c290f0910004b818ffd17de0228d0da" candidate_uuid: "d9a2464d933940528b5ed4a0d109527f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff3c26fdd0334804bc1c6051b2fea291" is_pre_election: true
I20260928 15:50:26.474666 24163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c290f0910004b818ffd17de0228d0da" candidate_uuid: "d9a2464d933940528b5ed4a0d109527f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7f81376474626bc8745c9970f066d" is_pre_election: true
I20260928 15:50:26.475090 23891 raft_consensus.cc:2454] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a2464d933940528b5ed4a0d109527f in term 0.
I20260928 15:50:26.475231 24163 raft_consensus.cc:2454] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9a2464d933940528b5ed4a0d109527f in term 0.
I20260928 15:50:26.475831 23962 leader_election.cc:302] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9a2464d933940528b5ed4a0d109527f, ff3c26fdd0334804bc1c6051b2fea291; no voters:
I20260928 15:50:26.476280 24235 raft_consensus.cc:2781] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:26.476438 24235 raft_consensus.cc:482] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:26.476553 24235 raft_consensus.cc:3037] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:26.478785 24235 raft_consensus.cc:504] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } }
I20260928 15:50:26.479465 24235 leader_election.cc:288] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [CANDIDATE]: Term 1 election: Requested vote from peers 07f7f81376474626bc8745c9970f066d (127.22.157.195:35291), ff3c26fdd0334804bc1c6051b2fea291 (127.22.157.193:38123)
I20260928 15:50:26.480077 24163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c290f0910004b818ffd17de0228d0da" candidate_uuid: "d9a2464d933940528b5ed4a0d109527f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f7f81376474626bc8745c9970f066d"
I20260928 15:50:26.480110 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c290f0910004b818ffd17de0228d0da" candidate_uuid: "d9a2464d933940528b5ed4a0d109527f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff3c26fdd0334804bc1c6051b2fea291"
I20260928 15:50:26.480269 24163 raft_consensus.cc:3037] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:26.480408 23891 raft_consensus.cc:3037] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:26.482677 24163 raft_consensus.cc:2454] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a2464d933940528b5ed4a0d109527f in term 1.
I20260928 15:50:26.483062 23961 leader_election.cc:302] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f7f81376474626bc8745c9970f066d, d9a2464d933940528b5ed4a0d109527f; no voters:
I20260928 15:50:26.483340 24235 raft_consensus.cc:2781] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:26.483326 23891 raft_consensus.cc:2454] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a2464d933940528b5ed4a0d109527f in term 1.
I20260928 15:50:26.484019 24235 raft_consensus.cc:686] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [term 1 LEADER]: Becoming Leader. State: Replica: d9a2464d933940528b5ed4a0d109527f, State: Running, Role: LEADER
I20260928 15:50:26.484517 24235 consensus_queue.cc:237] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } }
I20260928 15:50:26.489173 23743 catalog_manager.cc:5495] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f reported cstate change: term changed from 0 to 1, leader changed from <none> to d9a2464d933940528b5ed4a0d109527f (127.22.157.194). New cstate: current_term: 1 leader_uuid: "d9a2464d933940528b5ed4a0d109527f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2464d933940528b5ed4a0d109527f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07f7f81376474626bc8745c9970f066d" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff3c26fdd0334804bc1c6051b2fea291" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38123 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:50:26 dist-test-slave-1vw0 krb5kdc[23697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
W20260928 15:50:26.506770 23939 tablet.cc:2369] T 7c290f0910004b818ffd17de0228d0da P ff3c26fdd0334804bc1c6051b2fea291: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:26.513306 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.498347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.22.157.193:38123)
0928 15:50:26.498517 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.498522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.499045 (+ 523us) server_negotiation.cc:407] Connection header received
0928 15:50:26.499182 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.499186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.499256 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.499385 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:26.501059 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.501801 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.503115 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.503548 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.503639 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.507702 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:26.507712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:26.507727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:26.507763 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:26.510704 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.511351 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.511358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.511360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.511487 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.511816 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.511819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.511821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.512217 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:26.512434 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.512771 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.512945 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 15:50:26.517616 24075 tablet.cc:2369] T 7c290f0910004b818ffd17de0228d0da P d9a2464d933940528b5ed4a0d109527f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 15:50:26 dist-test-slave-1vw0 krb5kdc[23697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260928 15:50:26.545308 24210 tablet.cc:2369] T 7c290f0910004b818ffd17de0228d0da P 07f7f81376474626bc8745c9970f066d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 15:50:26 dist-test-slave-1vw0 krb5kdc[23697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610626, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:50:26.562980 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.548212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.22.157.193:38123)
0928 15:50:26.548462 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.548471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.548826 (+ 355us) server_negotiation.cc:407] Connection header received
0928 15:50:26.549033 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.549043 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.549109 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.549221 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:26.550569 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.551329 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.552611 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.553093 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.553231 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.557378 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:26.557383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:26.557387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:26.557424 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:26.560172 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.560891 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.560897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.560901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.561057 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.561494 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.561500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.561502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.561786 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:26.562086 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.562375 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.562641 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 15:50:26 dist-test-slave-1vw0 krb5kdc[23697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:50:26 dist-test-slave-1vw0 krb5kdc[23697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:50:26.613149 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.598341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.22.157.193:38123)
0928 15:50:26.598655 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.598662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.598878 (+ 216us) server_negotiation.cc:407] Connection header received
0928 15:50:26.599080 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.599085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.599160 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.599257 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:26.600962 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.601702 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.603006 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.603451 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.603573 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.607806 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:26.607811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:26.607815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:26.607875 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:26.610489 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.611312 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.611316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.611318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.611441 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.611760 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.611762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.611764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.612061 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:26.612261 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.612482 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.612706 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260928 15:50:26.614959 23850 tablet_service.cc:2908] Rejecting scan request for tablet 7c290f0910004b818ffd17de0228d0da: Uninitialized: clean time has not yet been initialized
I20260928 15:50:26.618790 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 23807
I20260928 15:50:26.633129 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.619729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.22.157.193:38123)
0928 15:50:26.619960 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.619966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.620265 (+ 299us) server_negotiation.cc:407] Connection header received
0928 15:50:26.620502 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.620507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.620584 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.620691 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:26.621974 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.622688 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.624099 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.624695 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.624921 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.627370 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:26.627375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:26.627379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:26.627419 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:26.630289 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.631075 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.631080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.631084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.631229 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.631610 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.631616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.631619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.631925 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:26.632225 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.632522 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.632805 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:50:26.633934 23933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:26.727226 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 23807
I20260928 15:50:26.744851 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 23944
Sep 28 15:50:26 dist-test-slave-1vw0 krb5kdc[23697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:50:26.759184 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.745599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.22.157.194:44773)
0928 15:50:26.745821 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.745829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.745910 (+ 81us) server_negotiation.cc:407] Connection header received
0928 15:50:26.746130 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.746135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.746207 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.746324 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:26.747487 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.748232 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.749600 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.750000 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.750136 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.754049 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:26.754063 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:26.754079 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:26.754125 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:26.756922 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.757545 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.757552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.757555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.757687 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.757996 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.757999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.758001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.758323 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:26.758553 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.758758 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.758933 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:50:26.759966 24069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:26.850327 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 23944
I20260928 15:50:26.868227 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 24079
Sep 28 15:50:26 dist-test-slave-1vw0 krb5kdc[23697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:50:26.883486 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:26.869180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.22.157.195:35291)
0928 15:50:26.869363 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:26.869368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:26.869387 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:50:26.869444 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:26.869448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:26.869510 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:26.869608 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:26.870799 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.871558 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.872873 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:26.873279 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:26.873376 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:26.877517 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:26.877532 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:26.877550 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:26.877592 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:26.881008 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.881648 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.881655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.881657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.881791 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:26.882138 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:26.882141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:26.882143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:26.882498 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:26.882717 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:26.882944 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 15:50:26.883114 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:50:26.884135 24203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:26.972644 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 24079
I20260928 15:50:26.989272 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 23711
I20260928 15:50:26.990442 23774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:27.099444 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 23711
2026-09-28T15:50:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4214 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:27 dist-test-slave-1vw0 krb5kdc[24257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:27 dist-test-slave-1vw0 krb5kdc[24257](info): set up 2 sockets
Sep 28 15:50:27 dist-test-slave-1vw0 krb5kdc[24257](info): commencing operation
krb5kdc: starting...
W20260928 15:50:28.189512 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:50:28 dist-test-slave-1vw0 krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:50:28.384536 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:34751
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:34751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:28.717614 24270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:28.717962 24270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:28.718055 24270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:28.726326 24270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:28.726451 24270 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:50:28.726492 24270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:28.726557 24270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:28.726605 24270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:28.726667 24270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:28.737041 24270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:34751
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:34751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:28.738325 24270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:28.739626 24270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:28.751829 24275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:28.751920 24278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:28.751897 24276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:28.752893 24270 server_base.cc:1034] running on GCE node
I20260928 15:50:28.753670 24270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:28.755154 24270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:28.756405 24270 hybrid_clock.cc:648] HybridClock initialized: now 1790610628756292 us; error 82 us; skew 500 ppm
I20260928 15:50:28.756861 24270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:28 dist-test-slave-1vw0 krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610628, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:28.760124 24270 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:50:28.761711 24270 webserver.cc:460] Webserver started at http://127.22.157.254:36087/ using document root <none> and password file <none>
I20260928 15:50:28.762482 24270 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:28.762612 24270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:28.762962 24270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:28.766088 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "552da18172a94b8b80bba489ab3aaad2"
format_stamp: "Formatted at 2026-09-28 15:50:28 on dist-test-slave-1vw0"
server_key: "d0158d932f9d55cc81c4a16ed27bc2ff"
server_key_iv: "34f8d24d33c5c06ab6d30da25470adc9"
server_key_version: "encryptionkey@0"
I20260928 15:50:28.766894 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "552da18172a94b8b80bba489ab3aaad2"
format_stamp: "Formatted at 2026-09-28 15:50:28 on dist-test-slave-1vw0"
server_key: "d0158d932f9d55cc81c4a16ed27bc2ff"
server_key_iv: "34f8d24d33c5c06ab6d30da25470adc9"
server_key_version: "encryptionkey@0"
I20260928 15:50:28.772351 24270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:50:28.776042 24286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:28.777575 24270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:28.777776 24270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "552da18172a94b8b80bba489ab3aaad2"
format_stamp: "Formatted at 2026-09-28 15:50:28 on dist-test-slave-1vw0"
server_key: "d0158d932f9d55cc81c4a16ed27bc2ff"
server_key_iv: "34f8d24d33c5c06ab6d30da25470adc9"
server_key_version: "encryptionkey@0"
I20260928 15:50:28.778020 24270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:28.813934 24270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:28.818255 24270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:28.818553 24270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:28.840577 24270 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:34751
I20260928 15:50:28.840605 24347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:34751 every 8 connection(s)
I20260928 15:50:28.842092 24270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:28.844463 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 24270
I20260928 15:50:28.844805 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:50:28.845475 23159 external_mini_cluster.cc:1428] Setting key fa3fa7b905b77fe6abee8b44f851e8d5
I20260928 15:50:28.846464 24348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:50:28 dist-test-slave-1vw0 krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:28.861368 24348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2: Bootstrap starting.
I20260928 15:50:28.865641 24348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:28.867053 24348 log.cc:824] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:28.870782 24348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2: No bootstrap required, opened a new log
I20260928 15:50:28.871850 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:28.847797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.22.157.254:34751)
0928 15:50:28.848606 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:28.848622 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:28.848674 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:50:28.850421 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:28.850450 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:28.851036 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:28.851532 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:28.852958 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:28.855154 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:28.856692 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:28.857847 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:28.858046 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:28.862480 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:28.862491 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:28.862511 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:28.862566 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:28.866777 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:28.867600 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:28.867618 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:28.867631 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:28.867853 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:28.868307 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:28.868311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:28.868313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:28.869164 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:28.869459 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:28.869940 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 15:50:28.870375 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":353,"threads_started":1}
I20260928 15:50:28.879439 24348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552da18172a94b8b80bba489ab3aaad2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34751 } }
I20260928 15:50:28.879858 24348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:28.879948 24348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552da18172a94b8b80bba489ab3aaad2, State: Initialized, Role: FOLLOWER
I20260928 15:50:28.880509 24348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552da18172a94b8b80bba489ab3aaad2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34751 } }
I20260928 15:50:28.880793 24348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:28.880977 24348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:28.881186 24348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:28.883551 24348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552da18172a94b8b80bba489ab3aaad2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34751 } }
I20260928 15:50:28.884045 24348 leader_election.cc:302] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 552da18172a94b8b80bba489ab3aaad2; no voters:
I20260928 15:50:28.884558 24348 leader_election.cc:288] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:28.884714 24353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:28.885452 24353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [term 1 LEADER]: Becoming Leader. State: Replica: 552da18172a94b8b80bba489ab3aaad2, State: Running, Role: LEADER
I20260928 15:50:28.886015 24353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552da18172a94b8b80bba489ab3aaad2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34751 } }
I20260928 15:50:28.886667 24348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:28.889245 24354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "552da18172a94b8b80bba489ab3aaad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552da18172a94b8b80bba489ab3aaad2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34751 } } }
I20260928 15:50:28.889726 24354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:28.890460 24355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 552da18172a94b8b80bba489ab3aaad2. Latest consensus state: current_term: 1 leader_uuid: "552da18172a94b8b80bba489ab3aaad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552da18172a94b8b80bba489ab3aaad2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34751 } } }
I20260928 15:50:28.890795 24355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:28.892499 24362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:28.897430 24362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:28.907132 24362 catalog_manager.cc:1347] Generated new cluster ID: ca0ae3a3c60d412dadbf90d883cfc407
I20260928 15:50:28.907286 24362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:28.929224 24362 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:28.930701 24362 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:28.939283 24362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2: Generated new TSK 0
I20260928 15:50:28.940181 24362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:50:29.001035 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:34751
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:29.340215 24375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:29.340564 24375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:29.340667 24375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:29.347925 24375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:29.348034 24375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:29.348120 24375 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:50:29.348201 24375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:50:29.358625 24375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.157.254:34751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:29.359933 24375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:29.361408 24375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:29.374413 24381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:29.374518 24380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:29.375164 24383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:29.375419 24375 server_base.cc:1034] running on GCE node
I20260928 15:50:29.376129 24375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:29.377173 24375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:29.378444 24375 hybrid_clock.cc:648] HybridClock initialized: now 1790610629378385 us; error 32 us; skew 500 ppm
I20260928 15:50:29.378770 24375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:29 dist-test-slave-1vw0 krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610629, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:29.382194 24375 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:50:29.383692 24375 webserver.cc:460] Webserver started at http://127.22.157.193:34559/ using document root <none> and password file <none>
I20260928 15:50:29.384356 24375 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:29.384451 24375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:29.384712 24375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:29.387807 24375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1"
format_stamp: "Formatted at 2026-09-28 15:50:29 on dist-test-slave-1vw0"
server_key: "1fd7ea0186d475a958a0808aade58a0b"
server_key_iv: "6a43ea59804964cd4f69011a1f8a86d8"
server_key_version: "encryptionkey@0"
I20260928 15:50:29.388581 24375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1"
format_stamp: "Formatted at 2026-09-28 15:50:29 on dist-test-slave-1vw0"
server_key: "1fd7ea0186d475a958a0808aade58a0b"
server_key_iv: "6a43ea59804964cd4f69011a1f8a86d8"
server_key_version: "encryptionkey@0"
I20260928 15:50:29.394279 24375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:50:29.398207 24390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:29.399756 24375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:50:29.399962 24375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1"
format_stamp: "Formatted at 2026-09-28 15:50:29 on dist-test-slave-1vw0"
server_key: "1fd7ea0186d475a958a0808aade58a0b"
server_key_iv: "6a43ea59804964cd4f69011a1f8a86d8"
server_key_version: "encryptionkey@0"
I20260928 15:50:29.400219 24375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:29.443779 24375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:29.448503 24375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:29.448871 24375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:29.451625 24375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:29.451711 24375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:29.451884 24375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:29.451946 24375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:50:29 dist-test-slave-1vw0 krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610629, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:29.479087 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:29.452813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:46517 (local address 127.22.157.254:34751)
0928 15:50:29.453332 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:29.453336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:29.454608 (+ 1272us) server_negotiation.cc:407] Connection header received
0928 15:50:29.456807 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:29.456814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:29.456907 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:29.457053 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:29.459857 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:29.460931 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:29.463766 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:29.464434 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:29.464622 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:29.470373 (+ 5751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:29.470384 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:29.470393 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:29.470455 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:29.474598 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:29.475851 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:29.475860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:29.475866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:29.476064 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:29.476706 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:29.476715 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:29.476740 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:29.477181 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:29.477361 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:29.478473 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0928 15:50:29.478706 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":296,"threads_started":1}
I20260928 15:50:29.480672 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:29.453551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:34751 (local address 127.22.157.193:46517)
0928 15:50:29.454420 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:29.454460 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:29.456440 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:29.457775 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:29.457784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:29.458628 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:29.459656 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:29.459679 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:29.461165 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:29.461169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:29.463596 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:29.463610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:29.464996 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:29.465002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:29.465317 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:29.466226 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:29.466270 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:29.470054 (+ 3784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:29.474838 (+ 4784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:29.474848 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:29.474871 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:29.475692 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:29.476272 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:29.476280 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:29.476290 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:29.476539 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:29.477370 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:29.477427 (+ 57us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:29.477967 (+ 540us) client_negotiation.cc:769] Sending connection context
0928 15:50:29.478385 (+ 418us) client_negotiation.cc:240] Negotiation successful
0928 15:50:29.478967 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":382,"threads_started":1}
I20260928 15:50:29.503000 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:29.487177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:50083 (local address 127.22.157.254:34751)
0928 15:50:29.487399 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:29.487408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:29.488005 (+ 597us) server_negotiation.cc:407] Connection header received
0928 15:50:29.488149 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:29.488157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:29.488294 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:29.488425 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:29.489568 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:29.491074 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:29.492501 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:29.493378 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:29.493537 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:29.496060 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:29.496069 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:29.496077 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:29.496131 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:29.499624 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:29.500619 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:29.500623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:29.500625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:29.500803 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:29.501301 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:29.501309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:29.501313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:29.501738 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:29.501908 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:29.502352 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 15:50:29.502589 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:50:29.503099 24478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:29.487164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:34751 (local address 127.22.157.193:50083)
0928 15:50:29.487900 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:29.487921 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:29.488043 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:29.488466 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:29.488469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:29.489009 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:29.489412 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:29.489422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:29.491256 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:29.491263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:29.492353 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:29.492366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:29.493615 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:29.493623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:29.493827 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:29.494610 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:29.494642 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:29.495815 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:29.499808 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:29.499815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:29.499825 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:29.500442 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:29.500966 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:29.500973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:29.500990 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:29.501177 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:29.501886 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:29.501890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:29.502091 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 15:50:29.502248 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 15:50:29.502613 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":377,"threads_started":1}
I20260928 15:50:29.511734 24375 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:37277
I20260928 15:50:29.511767 24515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:37277 every 8 connection(s)
I20260928 15:50:29.513027 24375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:50:29.513562 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 24375
I20260928 15:50:29.513818 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:50:29.514699 23159 external_mini_cluster.cc:1428] Setting key 35fdc02bacfe5f83728aaaa087cfa021
I20260928 15:50:29.515357 24516 heartbeater.cc:347] Connected to a master server at 127.22.157.254:34751
I20260928 15:50:29.515609 24516 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:29.516280 24516 heartbeater.cc:511] Master 127.22.157.254:34751 requested a full tablet report, sending...
I20260928 15:50:29.518393 24303 ts_manager.cc:194] Registered new tserver with Master: 5efc3b7be87f4f5eaadc5b4129eda1a1 (127.22.157.193:37277)
I20260928 15:50:29.519861 24303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:46517
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:50:29.560209 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:34751
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:29.901008 24520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:29.901355 24520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:29.901535 24520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:29.909061 24520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:29.909226 24520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:29.909389 24520 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:50:29.909534 24520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:50:29.919620 24520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.157.254:34751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:29.921095 24520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:29.922555 24520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:29.936326 24528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:29.936380 24526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:29.936363 24525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:29.937651 24520 server_base.cc:1034] running on GCE node
I20260928 15:50:29.938421 24520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:29.939733 24520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:29.941053 24520 hybrid_clock.cc:648] HybridClock initialized: now 1790610629940939 us; error 70 us; skew 500 ppm
I20260928 15:50:29.941411 24520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:29 dist-test-slave-1vw0 krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610629, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:29.944792 24520 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:50:29.946417 24520 webserver.cc:460] Webserver started at http://127.22.157.194:41407/ using document root <none> and password file <none>
I20260928 15:50:29.947140 24520 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:29.947233 24520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:29.947525 24520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:29.950927 24520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "37eb8ffce9dd49d4901316d7d338cbf4"
format_stamp: "Formatted at 2026-09-28 15:50:29 on dist-test-slave-1vw0"
server_key: "2af25c4e75652be014ba3b7d1de846a3"
server_key_iv: "348ce9aa97cf7ba7b8f73ef5caefdd9d"
server_key_version: "encryptionkey@0"
I20260928 15:50:29.951743 24520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "37eb8ffce9dd49d4901316d7d338cbf4"
format_stamp: "Formatted at 2026-09-28 15:50:29 on dist-test-slave-1vw0"
server_key: "2af25c4e75652be014ba3b7d1de846a3"
server_key_iv: "348ce9aa97cf7ba7b8f73ef5caefdd9d"
server_key_version: "encryptionkey@0"
I20260928 15:50:29.958073 24520 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 15:50:29.962044 24535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:29.963557 24520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:50:29.963704 24520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "37eb8ffce9dd49d4901316d7d338cbf4"
format_stamp: "Formatted at 2026-09-28 15:50:29 on dist-test-slave-1vw0"
server_key: "2af25c4e75652be014ba3b7d1de846a3"
server_key_iv: "348ce9aa97cf7ba7b8f73ef5caefdd9d"
server_key_version: "encryptionkey@0"
I20260928 15:50:29.963902 24520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:29.998615 24520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:30.003083 24520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:30.003317 24520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:30.006337 24520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:30.006458 24520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:30.006628 24520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:30.006708 24520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:50:30 dist-test-slave-1vw0 krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610629, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:30.033427 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.007226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:58943 (local address 127.22.157.254:34751)
0928 15:50:30.007716 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.007723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.009100 (+ 1377us) server_negotiation.cc:407] Connection header received
0928 15:50:30.011873 (+ 2773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.011883 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.012021 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.012167 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:30.014657 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.015772 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.019124 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.019727 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.019889 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.025277 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:30.025283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:30.025286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:30.025326 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:30.028480 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.029871 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.029877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.029882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.030071 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.030724 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.030730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.030734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.031294 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:30.031489 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.032321 (+ 832us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.032663 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
I20260928 15:50:30.033942 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.007964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:34751 (local address 127.22.157.194:58943)
0928 15:50:30.008903 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:30.008952 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:30.011397 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:30.012505 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:30.012532 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:30.013426 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:30.014450 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.014482 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.015996 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.016004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.018911 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.018937 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.020274 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.020286 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.020555 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.021450 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:30.021477 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:30.024913 (+ 3436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:30.028663 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:30.028674 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:30.028696 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:30.029683 (+ 987us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:30.030224 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:30.030227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:30.030230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:30.030455 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:30.031465 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:30.031471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:30.031880 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 15:50:30.032139 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 15:50:30.032557 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":406,"threads_started":1}
I20260928 15:50:30.053494 24520 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:44523
I20260928 15:50:30.053503 24657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:44523 every 8 connection(s)
I20260928 15:50:30.054812 24520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:50:30.056030 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.040653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:59195 (local address 127.22.157.254:34751)
0928 15:50:30.040915 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.040923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.041660 (+ 737us) server_negotiation.cc:407] Connection header received
0928 15:50:30.041876 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.041882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.041972 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.042074 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:30.043452 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.044379 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.046555 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.047202 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.047406 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.049856 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:30.049865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:30.049873 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:30.049926 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:30.053061 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.053931 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.053935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.053938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.054069 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.054483 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.054487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.054489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.054809 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:30.054968 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.055402 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.055640 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260928 15:50:30.056183 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.040788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:34751 (local address 127.22.157.194:59195)
0928 15:50:30.041531 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:30.041553 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:30.041746 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:30.042285 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:30.042291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:30.042959 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:30.043286 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.043296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.044531 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.044537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.046311 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.046336 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.047787 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.047794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.047928 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.048826 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:30.048845 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:30.049636 (+ 791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:30.053203 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:30.053208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:30.053212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:30.053792 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:30.054196 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:30.054201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:30.054205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:30.054360 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:30.054930 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:30.054936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:30.055150 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 15:50:30.055294 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 15:50:30.055705 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":455,"threads_started":1}
I20260928 15:50:30.057685 24658 heartbeater.cc:347] Connected to a master server at 127.22.157.254:34751
I20260928 15:50:30.058032 24658 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:30.058813 24658 heartbeater.cc:511] Master 127.22.157.254:34751 requested a full tablet report, sending...
I20260928 15:50:30.060230 24303 ts_manager.cc:194] Registered new tserver with Master: 37eb8ffce9dd49d4901316d7d338cbf4 (127.22.157.194:44523)
I20260928 15:50:30.060894 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 24520
I20260928 15:50:30.061091 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:50:30.061235 24303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:58943
I20260928 15:50:30.061599 23159 external_mini_cluster.cc:1428] Setting key 00d876645f4f01ca3e90115737c26c89
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:50:30.105369 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:34751
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:30.432615 24662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:30.432934 24662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:30.433117 24662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:30.441260 24662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:30.441373 24662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:30.441552 24662 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:50:30.441691 24662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:50:30.451818 24662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.157.254:34751
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:30.453210 24662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:30.454674 24662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:30.468669 24670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:30.468767 24668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:30.468395 24667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:30.469496 24662 server_base.cc:1034] running on GCE node
I20260928 15:50:30.470273 24662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:30.471315 24662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:30.472573 24662 hybrid_clock.cc:648] HybridClock initialized: now 1790610630472492 us; error 52 us; skew 500 ppm
I20260928 15:50:30.472965 24662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:30 dist-test-slave-1vw0 krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610630, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:30.476280 24662 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:50:30.477846 24662 webserver.cc:460] Webserver started at http://127.22.157.195:43575/ using document root <none> and password file <none>
I20260928 15:50:30.478579 24662 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:30.478710 24662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:30.479039 24662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:30.482199 24662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "c1cf98898121442c89df493930f2012e"
format_stamp: "Formatted at 2026-09-28 15:50:30 on dist-test-slave-1vw0"
server_key: "20ffe6f4802850433f1ee1d1472e2776"
server_key_iv: "be32a4a4ccb1b1e9768b9512bef93d5b"
server_key_version: "encryptionkey@0"
I20260928 15:50:30.483139 24662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "c1cf98898121442c89df493930f2012e"
format_stamp: "Formatted at 2026-09-28 15:50:30 on dist-test-slave-1vw0"
server_key: "20ffe6f4802850433f1ee1d1472e2776"
server_key_iv: "be32a4a4ccb1b1e9768b9512bef93d5b"
server_key_version: "encryptionkey@0"
I20260928 15:50:30.488443 24662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:50:30.492214 24677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:30.493806 24662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:30.493952 24662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "c1cf98898121442c89df493930f2012e"
format_stamp: "Formatted at 2026-09-28 15:50:30 on dist-test-slave-1vw0"
server_key: "20ffe6f4802850433f1ee1d1472e2776"
server_key_iv: "be32a4a4ccb1b1e9768b9512bef93d5b"
server_key_version: "encryptionkey@0"
I20260928 15:50:30.494158 24662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:30.512300 24662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:30.517647 24662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:30.517927 24662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:30.520610 24662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:30.520684 24662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:30.520854 24662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:30.520911 24662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:30.522732 24516 heartbeater.cc:503] Master 127.22.157.254:34751 was elected leader, sending a full tablet report...
Sep 28 15:50:30 dist-test-slave-1vw0 krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610630, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:30.545570 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.521357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:41335 (local address 127.22.157.254:34751)
0928 15:50:30.521576 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.521583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.522963 (+ 1380us) server_negotiation.cc:407] Connection header received
0928 15:50:30.525470 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.525476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.525572 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.525701 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:30.527905 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.529022 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.531776 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.532375 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.532541 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.537692 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:30.537700 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:30.537709 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:30.537746 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:30.541151 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.542606 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.542614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.542618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.542797 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.543395 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.543403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.543408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.543737 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:30.543901 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.544948 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.545193 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:50:30.547207 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.521880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:34751 (local address 127.22.157.195:41335)
0928 15:50:30.522747 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:30.522802 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:30.525125 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:30.525891 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:30.525923 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:30.526751 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:30.527709 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.527739 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.529263 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.529269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.531608 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.531624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.532636 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.532647 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.533029 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.533879 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:30.533920 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:30.537409 (+ 3489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:30.541373 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:30.541387 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:30.541409 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:30.542432 (+ 1023us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:30.542967 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:30.542978 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:30.542983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:30.543263 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:30.543876 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:30.543888 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:30.544422 (+ 534us) client_negotiation.cc:769] Sending connection context
0928 15:50:30.544855 (+ 433us) client_negotiation.cc:240] Negotiation successful
0928 15:50:30.545441 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":359,"threads_started":1}
I20260928 15:50:30.567155 24662 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:36571
I20260928 15:50:30.567188 24798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:36571 every 8 connection(s)
I20260928 15:50:30.568645 24662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:50:30.570240 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.554969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:45783 (local address 127.22.157.254:34751)
0928 15:50:30.555160 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.555165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.555501 (+ 336us) server_negotiation.cc:407] Connection header received
0928 15:50:30.555797 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.555804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.555896 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.556011 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:30.557280 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.558282 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.560409 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.561099 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.561252 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.563445 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:30.563452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:30.563459 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:30.563524 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:30.566871 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.567906 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.567911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.567926 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.568110 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.568491 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.568500 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.568503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.568896 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:30.569067 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.569504 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.569745 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:50:30.570299 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.554662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:34751 (local address 127.22.157.195:45783)
0928 15:50:30.555358 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:30.555386 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:30.555566 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:30.556050 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:30.556057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:30.556803 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:30.557128 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.557141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.558416 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.558424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.560234 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.560248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.561519 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.561524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.561657 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.562449 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:30.562467 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:30.563214 (+ 747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:30.567033 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:30.567049 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:30.567053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:30.567671 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:30.568232 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:30.568237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:30.568240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:30.568379 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:30.569043 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:30.569048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:30.569244 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 15:50:30.569410 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 15:50:30.569791 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":397,"threads_started":1}
I20260928 15:50:30.571214 24799 heartbeater.cc:347] Connected to a master server at 127.22.157.254:34751
I20260928 15:50:30.571530 24799 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:30.572625 24799 heartbeater.cc:511] Master 127.22.157.254:34751 requested a full tablet report, sending...
I20260928 15:50:30.574534 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 24662
I20260928 15:50:30.574605 24303 ts_manager.cc:194] Registered new tserver with Master: c1cf98898121442c89df493930f2012e (127.22.157.195:36571)
I20260928 15:50:30.574747 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:50:30.575194 23159 external_mini_cluster.cc:1428] Setting key 0ad5ccdeaa027a691534cbfb6d040d5c
I20260928 15:50:30.575506 24303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:41335
I20260928 15:50:30.578397 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:50:30 dist-test-slave-1vw0 krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610630, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:50:30 dist-test-slave-1vw0 krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610630, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:30.626408 24544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.612565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.22.157.254:34751)
0928 15:50:30.612828 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.612832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.613110 (+ 278us) server_negotiation.cc:407] Connection header received
0928 15:50:30.613280 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.613284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.613364 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.613461 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:30.614848 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.615573 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.616869 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.617275 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.617398 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.621251 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:30.621255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:30.621258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:30.621290 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:30.623693 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.624332 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.624337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.624339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.624459 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:30.624903 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:30.624910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:30.624918 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:30.625220 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:30.625485 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.625805 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.626047 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:50:30.630759 24303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:50:30.637874 24303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:50:30.666733 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.651582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.22.157.195:36571)
0928 15:50:30.658102 (+ 6520us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.658111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.658149 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:50:30.658251 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.658260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.658519 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.658754 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:30.659785 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.661407 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.663836 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.665344 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.665959 (+ 615us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.666143 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.666216 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.666381 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6346,"thread_start_us":6138,"threads_started":1}
I20260928 15:50:30.666788 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.650931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:36571 (local address 127.0.0.1:51854)
0928 15:50:30.652876 (+ 1945us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:30.652899 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:30.653041 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:30.658803 (+ 5762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:30.658811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:30.658852 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:30.659508 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.659522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.661556 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.661561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.663688 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.663698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.665579 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.665586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.665818 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.665958 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 15:50:30.666084 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 15:50:30.666360 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1754,"mutex_wait_us":107,"thread_start_us":8223,"threads_started":1}
I20260928 15:50:30.667207 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.651359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.22.157.193:37277)
0928 15:50:30.657235 (+ 5876us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.657246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.657282 (+ 36us) server_negotiation.cc:407] Connection header received
0928 15:50:30.657396 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.657403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.657710 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.658714 (+ 1004us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:30.658952 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.660546 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.663601 (+ 3055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.665293 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.665477 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.665676 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.666502 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.666669 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5689,"thread_start_us":5562,"threads_started":1}
I20260928 15:50:30.667623 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.650711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37277 (local address 127.0.0.1:56368)
0928 15:50:30.651692 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:30.651738 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:30.651899 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:30.658013 (+ 6114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:30.658022 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:30.658057 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:30.658484 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.658498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.660716 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.660746 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.663421 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.663439 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.665500 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.665510 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.665805 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.665957 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 15:50:30.666084 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:50:30.666360 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":189,"thread_start_us":810,"threads_started":1}
I20260928 15:50:30.667851 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.652133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:44523 (local address 127.0.0.1:43002)
0928 15:50:30.659511 (+ 7378us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:30.659532 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:30.659758 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:30.660561 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:30.660565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:30.660614 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:30.661175 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.661191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.662904 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.662910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.664931 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.664948 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.666929 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.666937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.667114 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.667151 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:50:30.667254 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:50:30.667394 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7218,"mutex_wait_us":30,"thread_start_us":4588,"threads_started":1}
I20260928 15:50:30.668222 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.652782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.22.157.194:44523)
0928 15:50:30.657537 (+ 4755us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.657546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.659647 (+ 2101us) server_negotiation.cc:407] Connection header received
0928 15:50:30.659940 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.659948 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.660186 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.660337 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:30.661339 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.662766 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.665086 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.666425 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.666614 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.666789 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.667327 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.667496 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4593,"thread_start_us":4641,"threads_started":1}
I20260928 15:50:30.671370 24728 tablet_service.cc:1467] Processing CreateTablet for tablet 88bfed009fb24596a4c21c3111bd5f7e (DEFAULT_TABLE table=test-table [id=765865ca80594e009fa8846c3cd14d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:30.671377 24444 tablet_service.cc:1467] Processing CreateTablet for tablet 88bfed009fb24596a4c21c3111bd5f7e (DEFAULT_TABLE table=test-table [id=765865ca80594e009fa8846c3cd14d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:30.672537 24444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bfed009fb24596a4c21c3111bd5f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:30.672545 24728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bfed009fb24596a4c21c3111bd5f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:30.675829 24587 tablet_service.cc:1467] Processing CreateTablet for tablet 88bfed009fb24596a4c21c3111bd5f7e (DEFAULT_TABLE table=test-table [id=765865ca80594e009fa8846c3cd14d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:30.677260 24587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88bfed009fb24596a4c21c3111bd5f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:30.684643 24818 tablet_bootstrap.cc:492] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: Bootstrap starting.
I20260928 15:50:30.687252 24819 tablet_bootstrap.cc:492] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: Bootstrap starting.
I20260928 15:50:30.689926 24818 tablet_bootstrap.cc:654] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:30.690825 24819 tablet_bootstrap.cc:654] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:30.691282 24818 log.cc:824] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:30.691318 24820 tablet_bootstrap.cc:492] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: Bootstrap starting.
I20260928 15:50:30.692366 24819 log.cc:824] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:30.694600 24820 tablet_bootstrap.cc:654] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:30.694856 24818 tablet_bootstrap.cc:492] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: No bootstrap required, opened a new log
I20260928 15:50:30.695147 24818 ts_tablet_manager.cc:1397] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260928 15:50:30.695405 24819 tablet_bootstrap.cc:492] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: No bootstrap required, opened a new log
I20260928 15:50:30.695662 24819 ts_tablet_manager.cc:1397] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 15:50:30.695703 24820 log.cc:824] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:30.706053 24820 tablet_bootstrap.cc:492] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: No bootstrap required, opened a new log
I20260928 15:50:30.706384 24820 ts_tablet_manager.cc:1397] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260928 15:50:30.706777 24819 raft_consensus.cc:348] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } }
I20260928 15:50:30.707247 24819 raft_consensus.cc:374] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:30.707352 24819 raft_consensus.cc:729] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cf98898121442c89df493930f2012e, State: Initialized, Role: FOLLOWER
I20260928 15:50:30.706777 24818 raft_consensus.cc:348] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } }
I20260928 15:50:30.707687 24818 raft_consensus.cc:374] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:30.707794 24818 raft_consensus.cc:729] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5efc3b7be87f4f5eaadc5b4129eda1a1, State: Initialized, Role: FOLLOWER
I20260928 15:50:30.708148 24819 consensus_queue.cc:260] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } }
I20260928 15:50:30.708351 24818 consensus_queue.cc:260] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } }
I20260928 15:50:30.710599 24818 ts_tablet_manager.cc:1428] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260928 15:50:30.714684 24799 heartbeater.cc:503] Master 127.22.157.254:34751 was elected leader, sending a full tablet report...
I20260928 15:50:30.715402 24819 ts_tablet_manager.cc:1428] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: Time spent starting tablet: real 0.020s user 0.011s sys 0.009s
I20260928 15:50:30.716754 24820 raft_consensus.cc:348] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } }
I20260928 15:50:30.717248 24820 raft_consensus.cc:374] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:30.717365 24820 raft_consensus.cc:729] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eb8ffce9dd49d4901316d7d338cbf4, State: Initialized, Role: FOLLOWER
I20260928 15:50:30.717898 24820 consensus_queue.cc:260] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } }
I20260928 15:50:30.719048 24658 heartbeater.cc:503] Master 127.22.157.254:34751 was elected leader, sending a full tablet report...
I20260928 15:50:30.719547 24820 ts_tablet_manager.cc:1428] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
W20260928 15:50:30.765692 24517 tablet.cc:2369] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:30.771306 24824 raft_consensus.cc:482] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:30.771682 24824 raft_consensus.cc:504] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } }
I20260928 15:50:30.773389 24824 leader_election.cc:288] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5efc3b7be87f4f5eaadc5b4129eda1a1 (127.22.157.193:37277), 37eb8ffce9dd49d4901316d7d338cbf4 (127.22.157.194:44523)
I20260928 15:50:30.779767 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.773815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:50653 (local address 127.22.157.193:37277)
0928 15:50:30.774018 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.774025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.774189 (+ 164us) server_negotiation.cc:407] Connection header received
0928 15:50:30.774421 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.774426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.774516 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.774632 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:30.775115 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.776038 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.777843 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.778842 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.778960 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.779020 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.779356 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.779502 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:50:30.779783 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.773702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37277 (local address 127.22.157.195:50653)
0928 15:50:30.774048 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:30.774075 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:30.774263 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:30.774692 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:30.774697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:30.774744 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:30.774958 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.774972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.776178 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.776183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.777693 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.777705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.778998 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.779004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.779197 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.779236 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:50:30.779338 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:50:30.779502 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":1}
I20260928 15:50:30.780501 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.773815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:50611 (local address 127.22.157.194:44523)
0928 15:50:30.774007 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.774013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.774682 (+ 669us) server_negotiation.cc:407] Connection header received
0928 15:50:30.774835 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.774841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.774924 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.775022 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:30.775645 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.776535 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.778428 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.779393 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.779540 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.779605 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.780106 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.780233 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:50:30.780840 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bfed009fb24596a4c21c3111bd5f7e" candidate_uuid: "c1cf98898121442c89df493930f2012e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" is_pre_election: true
I20260928 15:50:30.781183 24827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.773790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:44523 (local address 127.22.157.195:50611)
0928 15:50:30.774567 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:30.774585 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:30.774704 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:30.775063 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:30.775068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:30.775093 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:30.775466 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.775476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.776665 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.776668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.778273 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:30.778286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.779721 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.779727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:30.779909 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.779949 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 15:50:30.780757 (+ 808us) client_negotiation.cc:240] Negotiation successful
0928 15:50:30.780875 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":414,"threads_started":1}
I20260928 15:50:30.781401 24464 raft_consensus.cc:2454] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1cf98898121442c89df493930f2012e in term 0.
I20260928 15:50:30.782148 24680 leader_election.cc:302] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5efc3b7be87f4f5eaadc5b4129eda1a1, c1cf98898121442c89df493930f2012e; no voters:
I20260928 15:50:30.782157 24607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bfed009fb24596a4c21c3111bd5f7e" candidate_uuid: "c1cf98898121442c89df493930f2012e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" is_pre_election: true
I20260928 15:50:30.782572 24824 raft_consensus.cc:2781] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:30.782706 24824 raft_consensus.cc:482] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:30.782687 24607 raft_consensus.cc:2454] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1cf98898121442c89df493930f2012e in term 0.
I20260928 15:50:30.782829 24824 raft_consensus.cc:3037] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:30.785311 24824 raft_consensus.cc:504] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } }
I20260928 15:50:30.785961 24824 leader_election.cc:288] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [CANDIDATE]: Term 1 election: Requested vote from peers 5efc3b7be87f4f5eaadc5b4129eda1a1 (127.22.157.193:37277), 37eb8ffce9dd49d4901316d7d338cbf4 (127.22.157.194:44523)
I20260928 15:50:30.786401 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bfed009fb24596a4c21c3111bd5f7e" candidate_uuid: "c1cf98898121442c89df493930f2012e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1"
I20260928 15:50:30.786517 24607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88bfed009fb24596a4c21c3111bd5f7e" candidate_uuid: "c1cf98898121442c89df493930f2012e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37eb8ffce9dd49d4901316d7d338cbf4"
I20260928 15:50:30.786628 24464 raft_consensus.cc:3037] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:30.786760 24607 raft_consensus.cc:3037] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:30.789088 24464 raft_consensus.cc:2454] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1cf98898121442c89df493930f2012e in term 1.
I20260928 15:50:30.789458 24607 raft_consensus.cc:2454] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1cf98898121442c89df493930f2012e in term 1.
I20260928 15:50:30.789485 24680 leader_election.cc:302] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5efc3b7be87f4f5eaadc5b4129eda1a1, c1cf98898121442c89df493930f2012e; no voters:
I20260928 15:50:30.789808 24824 raft_consensus.cc:2781] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:30.790261 24824 raft_consensus.cc:686] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 1 LEADER]: Becoming Leader. State: Replica: c1cf98898121442c89df493930f2012e, State: Running, Role: LEADER
I20260928 15:50:30.790768 24824 consensus_queue.cc:237] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } }
I20260928 15:50:30.796099 24302 catalog_manager.cc:5495] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e reported cstate change: term changed from 0 to 1, leader changed from <none> to c1cf98898121442c89df493930f2012e (127.22.157.195). New cstate: current_term: 1 leader_uuid: "c1cf98898121442c89df493930f2012e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } health_report { overall_health: UNKNOWN } } }
W20260928 15:50:30.808089 24659 tablet.cc:2369] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:50:30.826519 24800 tablet.cc:2369] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:30.829401 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:30.823633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.22.157.195:36571)
0928 15:50:30.823814 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:30.823819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:30.823893 (+ 74us) server_negotiation.cc:407] Connection header received
0928 15:50:30.824168 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:30.824172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:30.824247 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:30.824366 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:30.824845 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.825599 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.827366 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:30.827790 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:30.827881 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:30.828334 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:30.828911 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:30.829027 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:30.829144 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 15:50:30.829207 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:50:30.837715 24464 raft_consensus.cc:1261] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Refusing update from remote peer c1cf98898121442c89df493930f2012e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:30.837715 24607 raft_consensus.cc:1261] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 FOLLOWER]: Refusing update from remote peer c1cf98898121442c89df493930f2012e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:30.838810 24828 consensus_queue.cc:1035] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:30.839205 24824 consensus_queue.cc:1035] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [LEADER]: Connected to new peer: Peer: permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:30.868862 24833 mvcc.cc:205] Tried to move back new op lower bound from 7334341143899561984 to 7334341143723220992. Current Snapshot: MvccSnapshot[applied={T|T < 7334341143899561984}]
I20260928 15:50:30.878947 24302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:30.879246 24302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:30.883038 24302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2: Sending DeleteTablet for 3 replicas of tablet 88bfed009fb24596a4c21c3111bd5f7e
I20260928 15:50:30.884590 24444 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bfed009fb24596a4c21c3111bd5f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:30 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:56368
I20260928 15:50:30.884583 24587 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bfed009fb24596a4c21c3111bd5f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:30 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:43002
I20260928 15:50:30.885600 24842 tablet_replica.cc:330] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: stopping tablet replica
I20260928 15:50:30.885882 24728 tablet_service.cc:1514] Processing DeleteTablet for tablet 88bfed009fb24596a4c21c3111bd5f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:30 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:51854
I20260928 15:50:30.885913 24302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 15:50:30.886246 24842 raft_consensus.cc:2229] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 15:50:30.886770 24302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:50:30.887246 24842 raft_consensus.cc:2258] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:30.888857 24844 tablet_replica.cc:330] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: stopping tablet replica
I20260928 15:50:30.889072 24843 tablet_replica.cc:330] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: stopping tablet replica
I20260928 15:50:30.889315 24844 raft_consensus.cc:2229] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:50:30.890056 24844 raft_consensus.cc:2258] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:30.890079 24843 raft_consensus.cc:2229] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:30.890805 24843 raft_consensus.cc:2258] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:30.892680 24843 ts_tablet_manager.cc:1905] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:30.895157 24842 ts_tablet_manager.cc:1905] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:30.895488 24444 tablet_service.cc:1467] Processing CreateTablet for tablet 379bb0e741554ef78eaddd6af9cea7f6 (DEFAULT_TABLE table=test-table [id=e26ce3a982b74ad99dd728e14e5d4a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:30.896004 24444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379bb0e741554ef78eaddd6af9cea7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:30.895922 24587 tablet_service.cc:1467] Processing CreateTablet for tablet 379bb0e741554ef78eaddd6af9cea7f6 (DEFAULT_TABLE table=test-table [id=e26ce3a982b74ad99dd728e14e5d4a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:30.896346 24587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379bb0e741554ef78eaddd6af9cea7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:30.901108 24818 tablet_bootstrap.cc:492] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: Bootstrap starting.
I20260928 15:50:30.902907 24818 tablet_bootstrap.cc:654] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:30.904139 24820 tablet_bootstrap.cc:492] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: Bootstrap starting.
I20260928 15:50:30.904134 24844 ts_tablet_manager.cc:1905] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:30.904263 24728 tablet_service.cc:1467] Processing CreateTablet for tablet 379bb0e741554ef78eaddd6af9cea7f6 (DEFAULT_TABLE table=test-table [id=e26ce3a982b74ad99dd728e14e5d4a93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:30.904888 24728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379bb0e741554ef78eaddd6af9cea7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:30.905794 24820 tablet_bootstrap.cc:654] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:30.909698 24842 ts_tablet_manager.cc:1918] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:50:30.909901 24842 log.cc:1196] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/88bfed009fb24596a4c21c3111bd5f7e
I20260928 15:50:30.910542 24842 ts_tablet_manager.cc:1939] T 88bfed009fb24596a4c21c3111bd5f7e P 5efc3b7be87f4f5eaadc5b4129eda1a1: Deleting consensus metadata
I20260928 15:50:30.911855 24819 tablet_bootstrap.cc:492] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: Bootstrap starting.
I20260928 15:50:30.912248 24844 ts_tablet_manager.cc:1918] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:50:30.912489 24844 log.cc:1196] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/88bfed009fb24596a4c21c3111bd5f7e
I20260928 15:50:30.912613 24289 catalog_manager.cc:4841] TS 5efc3b7be87f4f5eaadc5b4129eda1a1 (127.22.157.193:37277): tablet 88bfed009fb24596a4c21c3111bd5f7e (table test-table [id=765865ca80594e009fa8846c3cd14d82]) successfully deleted
I20260928 15:50:30.913241 24844 ts_tablet_manager.cc:1939] T 88bfed009fb24596a4c21c3111bd5f7e P c1cf98898121442c89df493930f2012e: Deleting consensus metadata
I20260928 15:50:30.913830 24819 tablet_bootstrap.cc:654] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:30.915355 24290 catalog_manager.cc:4841] TS c1cf98898121442c89df493930f2012e (127.22.157.195:36571): tablet 88bfed009fb24596a4c21c3111bd5f7e (table test-table [id=765865ca80594e009fa8846c3cd14d82]) successfully deleted
I20260928 15:50:30.919117 24843 ts_tablet_manager.cc:1918] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:50:30.919315 24843 log.cc:1196] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/88bfed009fb24596a4c21c3111bd5f7e
I20260928 15:50:30.919867 24843 ts_tablet_manager.cc:1939] T 88bfed009fb24596a4c21c3111bd5f7e P 37eb8ffce9dd49d4901316d7d338cbf4: Deleting consensus metadata
I20260928 15:50:30.921897 24288 catalog_manager.cc:4841] TS 37eb8ffce9dd49d4901316d7d338cbf4 (127.22.157.194:44523): tablet 88bfed009fb24596a4c21c3111bd5f7e (table test-table [id=765865ca80594e009fa8846c3cd14d82]) successfully deleted
I20260928 15:50:30.926483 24819 tablet_bootstrap.cc:492] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: No bootstrap required, opened a new log
I20260928 15:50:30.926677 24819 ts_tablet_manager.cc:1397] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20260928 15:50:30.926988 24820 tablet_bootstrap.cc:492] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: No bootstrap required, opened a new log
I20260928 15:50:30.927538 24820 ts_tablet_manager.cc:1397] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.010s
I20260928 15:50:30.927632 24819 raft_consensus.cc:348] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:30.927891 24819 raft_consensus.cc:374] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:30.928004 24819 raft_consensus.cc:729] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cf98898121442c89df493930f2012e, State: Initialized, Role: FOLLOWER
I20260928 15:50:30.928340 24819 consensus_queue.cc:260] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:30.928383 24820 raft_consensus.cc:348] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:30.928606 24820 raft_consensus.cc:374] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:30.929188 24819 ts_tablet_manager.cc:1428] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 15:50:30.929500 24820 raft_consensus.cc:729] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eb8ffce9dd49d4901316d7d338cbf4, State: Initialized, Role: FOLLOWER
I20260928 15:50:30.929662 24818 tablet_bootstrap.cc:492] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: No bootstrap required, opened a new log
I20260928 15:50:30.929857 24818 ts_tablet_manager.cc:1397] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.007s
I20260928 15:50:30.930511 24818 raft_consensus.cc:348] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:30.930713 24818 raft_consensus.cc:374] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:30.930835 24818 raft_consensus.cc:729] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5efc3b7be87f4f5eaadc5b4129eda1a1, State: Initialized, Role: FOLLOWER
I20260928 15:50:30.931123 24818 consensus_queue.cc:260] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:30.931823 24818 ts_tablet_manager.cc:1428] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:50:30.931304 24820 consensus_queue.cc:260] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:30.932762 24820 ts_tablet_manager.cc:1428] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260928 15:50:31.061684 24826 raft_consensus.cc:482] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:31.061935 24826 raft_consensus.cc:504] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.063910 24826 leader_election.cc:288] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5efc3b7be87f4f5eaadc5b4129eda1a1 (127.22.157.193:37277), c1cf98898121442c89df493930f2012e (127.22.157.195:36571)
I20260928 15:50:31.083078 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.065278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:33905 (local address 127.22.157.193:37277)
0928 15:50:31.065461 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.065467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.074519 (+ 9052us) server_negotiation.cc:407] Connection header received
0928 15:50:31.074739 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.074746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.074832 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.074947 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:31.075998 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.077212 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.080190 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.081565 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.081714 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.081790 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.082613 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.082761 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:50:31.083276 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.064925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:39175 (local address 127.22.157.195:36571)
0928 15:50:31.065123 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.065130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.075456 (+ 10326us) server_negotiation.cc:407] Connection header received
0928 15:50:31.075723 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.075729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.075821 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.075880 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:31.076776 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.077820 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.080396 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.081812 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.081934 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.082007 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.082831 (+ 824us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.082978 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:50:31.084074 24848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.064137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37277 (local address 127.22.157.194:33905)
0928 15:50:31.074364 (+ 10227us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:31.074390 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:31.074594 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:31.075337 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:31.075346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:31.075377 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:31.075795 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.075808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.077432 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.077439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.079979 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.079994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.082090 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.082096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.082295 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.082440 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 15:50:31.083483 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0928 15:50:31.083637 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10062,"mutex_wait_us":165,"thread_start_us":10561,"threads_started":1}
I20260928 15:50:31.084384 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.064387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:36571 (local address 127.22.157.194:39175)
0928 15:50:31.075356 (+ 10969us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:31.075374 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:31.075571 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:31.076131 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:31.076135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:31.076155 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:31.076547 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.076565 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.077995 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.078000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.080195 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.080213 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.082301 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.082308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.082611 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.082645 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:50:31.083753 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0928 15:50:31.083949 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10838,"mutex_wait_us":32,"thread_start_us":10711,"threads_started":1}
I20260928 15:50:31.084758 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379bb0e741554ef78eaddd6af9cea7f6" candidate_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" is_pre_election: true
I20260928 15:50:31.085034 24464 raft_consensus.cc:2454] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eb8ffce9dd49d4901316d7d338cbf4 in term 0.
I20260928 15:50:31.085451 24748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379bb0e741554ef78eaddd6af9cea7f6" candidate_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cf98898121442c89df493930f2012e" is_pre_election: true
I20260928 15:50:31.085541 24538 leader_election.cc:302] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37eb8ffce9dd49d4901316d7d338cbf4, 5efc3b7be87f4f5eaadc5b4129eda1a1; no voters:
I20260928 15:50:31.085891 24748 raft_consensus.cc:2454] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eb8ffce9dd49d4901316d7d338cbf4 in term 0.
I20260928 15:50:31.085976 24826 raft_consensus.cc:2781] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:31.086107 24826 raft_consensus.cc:482] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:31.086195 24826 raft_consensus.cc:3037] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:31.089650 24826 raft_consensus.cc:504] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.090498 24826 leader_election.cc:288] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [CANDIDATE]: Term 1 election: Requested vote from peers 5efc3b7be87f4f5eaadc5b4129eda1a1 (127.22.157.193:37277), c1cf98898121442c89df493930f2012e (127.22.157.195:36571)
I20260928 15:50:31.091161 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379bb0e741554ef78eaddd6af9cea7f6" candidate_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1"
I20260928 15:50:31.091228 24748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379bb0e741554ef78eaddd6af9cea7f6" candidate_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cf98898121442c89df493930f2012e"
I20260928 15:50:31.091459 24748 raft_consensus.cc:3037] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:31.091445 24464 raft_consensus.cc:3037] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:31.093768 24464 raft_consensus.cc:2454] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eb8ffce9dd49d4901316d7d338cbf4 in term 1.
I20260928 15:50:31.094395 24538 leader_election.cc:302] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37eb8ffce9dd49d4901316d7d338cbf4, 5efc3b7be87f4f5eaadc5b4129eda1a1; no voters:
I20260928 15:50:31.095552 24826 raft_consensus.cc:2781] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:31.095814 24748 raft_consensus.cc:2454] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eb8ffce9dd49d4901316d7d338cbf4 in term 1.
I20260928 15:50:31.096095 24826 raft_consensus.cc:686] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 LEADER]: Becoming Leader. State: Replica: 37eb8ffce9dd49d4901316d7d338cbf4, State: Running, Role: LEADER
I20260928 15:50:31.096562 24826 consensus_queue.cc:237] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.100559 24303 catalog_manager.cc:5495] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37eb8ffce9dd49d4901316d7d338cbf4 (127.22.157.194). New cstate: current_term: 1 leader_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:50:31 dist-test-slave-1vw0 krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610628, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:31.179473 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.163191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:34751 (local address 127.0.0.1:45862)
0928 15:50:31.164125 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:31.164147 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:31.164278 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:31.164658 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:31.164661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:31.165509 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:31.165785 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.165796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.167450 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.167454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.168693 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.168706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.169727 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.169731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.169864 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.170616 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:31.170635 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:31.172919 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:31.176934 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:31.176948 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:31.176955 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:31.177387 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:31.177864 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:31.177867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:31.177870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:31.177988 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:31.178680 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:31.178686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:31.178831 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 15:50:31.178967 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:50:31.179182 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":567,"threads_started":1}
I20260928 15:50:31.179862 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.163178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.22.157.254:34751)
0928 15:50:31.163867 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.163873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.164241 (+ 368us) server_negotiation.cc:407] Connection header received
0928 15:50:31.164394 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.164401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.164485 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.164615 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:31.165986 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.167305 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.168890 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.169546 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.169693 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.173157 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:31.173165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:31.173170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:31.173221 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:31.176766 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.177539 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.177545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.177548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.177737 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.178125 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.178131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.178135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.178543 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:31.178711 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.179141 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.179393 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":309,"threads_started":1}
I20260928 15:50:31.186115 24303 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:45862:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:50:31.187453 24303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:50:31.195219 24444 tablet_service.cc:1467] Processing CreateTablet for tablet 79b6f55f777041da8e1d77b423bed8ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a769eb95b41344c5b9c3ea6856c48d1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:50:31.195672 24728 tablet_service.cc:1467] Processing CreateTablet for tablet 79b6f55f777041da8e1d77b423bed8ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a769eb95b41344c5b9c3ea6856c48d1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:50:31.195896 24444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b6f55f777041da8e1d77b423bed8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:31.195854 24587 tablet_service.cc:1467] Processing CreateTablet for tablet 79b6f55f777041da8e1d77b423bed8ab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a769eb95b41344c5b9c3ea6856c48d1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:50:31.196286 24728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b6f55f777041da8e1d77b423bed8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:31.196441 24587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79b6f55f777041da8e1d77b423bed8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:31.204074 24818 tablet_bootstrap.cc:492] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: Bootstrap starting.
I20260928 15:50:31.205610 24818 tablet_bootstrap.cc:654] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:31.206691 24820 tablet_bootstrap.cc:492] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: Bootstrap starting.
I20260928 15:50:31.208043 24820 tablet_bootstrap.cc:654] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:31.209604 24819 tablet_bootstrap.cc:492] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: Bootstrap starting.
I20260928 15:50:31.211802 24818 tablet_bootstrap.cc:492] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: No bootstrap required, opened a new log
I20260928 15:50:31.212096 24818 ts_tablet_manager.cc:1397] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260928 15:50:31.212155 24819 tablet_bootstrap.cc:654] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:31.213621 24818 raft_consensus.cc:348] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.213985 24818 raft_consensus.cc:374] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:31.214159 24818 raft_consensus.cc:729] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5efc3b7be87f4f5eaadc5b4129eda1a1, State: Initialized, Role: FOLLOWER
I20260928 15:50:31.214557 24818 consensus_queue.cc:260] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.215973 24818 ts_tablet_manager.cc:1428] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 15:50:31.215884 24825 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.216401 24825 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:31.218535 24820 tablet_bootstrap.cc:492] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: No bootstrap required, opened a new log
I20260928 15:50:31.218724 24820 ts_tablet_manager.cc:1397] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260928 15:50:31.219523 24819 tablet_bootstrap.cc:492] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: No bootstrap required, opened a new log
I20260928 15:50:31.219722 24819 ts_tablet_manager.cc:1397] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260928 15:50:31.219722 24820 raft_consensus.cc:348] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.219977 24820 raft_consensus.cc:374] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:31.220082 24820 raft_consensus.cc:729] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eb8ffce9dd49d4901316d7d338cbf4, State: Initialized, Role: FOLLOWER
I20260928 15:50:31.220331 24820 consensus_queue.cc:260] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.220566 24819 raft_consensus.cc:348] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.220832 24819 raft_consensus.cc:374] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:31.220916 24819 raft_consensus.cc:729] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cf98898121442c89df493930f2012e, State: Initialized, Role: FOLLOWER
I20260928 15:50:31.221022 24820 ts_tablet_manager.cc:1428] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 15:50:31.220863 24826 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.221204 24826 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:31.221199 24819 consensus_queue.cc:260] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.221585 24828 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.221941 24828 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:31.221813 24819 ts_tablet_manager.cc:1428] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:50:31.339901 24826 raft_consensus.cc:482] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:31.340147 24826 raft_consensus.cc:504] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.340854 24826 leader_election.cc:288] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5efc3b7be87f4f5eaadc5b4129eda1a1 (127.22.157.193:37277), c1cf98898121442c89df493930f2012e (127.22.157.195:36571)
I20260928 15:50:31.341427 24748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b6f55f777041da8e1d77b423bed8ab" candidate_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cf98898121442c89df493930f2012e" is_pre_election: true
I20260928 15:50:31.341466 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b6f55f777041da8e1d77b423bed8ab" candidate_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" is_pre_election: true
I20260928 15:50:31.341720 24748 raft_consensus.cc:2454] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eb8ffce9dd49d4901316d7d338cbf4 in term 0.
I20260928 15:50:31.341749 24464 raft_consensus.cc:2454] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37eb8ffce9dd49d4901316d7d338cbf4 in term 0.
I20260928 15:50:31.342233 24539 leader_election.cc:302] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37eb8ffce9dd49d4901316d7d338cbf4, c1cf98898121442c89df493930f2012e; no voters:
I20260928 15:50:31.342537 24826 raft_consensus.cc:2781] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:31.342651 24826 raft_consensus.cc:482] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:31.342815 24826 raft_consensus.cc:3037] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:31.344899 24826 raft_consensus.cc:504] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.345544 24826 leader_election.cc:288] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [CANDIDATE]: Term 1 election: Requested vote from peers 5efc3b7be87f4f5eaadc5b4129eda1a1 (127.22.157.193:37277), c1cf98898121442c89df493930f2012e (127.22.157.195:36571)
I20260928 15:50:31.345968 24464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b6f55f777041da8e1d77b423bed8ab" candidate_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1"
I20260928 15:50:31.346055 24748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79b6f55f777041da8e1d77b423bed8ab" candidate_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cf98898121442c89df493930f2012e"
I20260928 15:50:31.346239 24464 raft_consensus.cc:3037] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:31.346299 24748 raft_consensus.cc:3037] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:31.349395 24464 raft_consensus.cc:2454] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eb8ffce9dd49d4901316d7d338cbf4 in term 1.
I20260928 15:50:31.349395 24748 raft_consensus.cc:2454] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37eb8ffce9dd49d4901316d7d338cbf4 in term 1.
I20260928 15:50:31.349879 24539 leader_election.cc:302] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37eb8ffce9dd49d4901316d7d338cbf4, 5efc3b7be87f4f5eaadc5b4129eda1a1, c1cf98898121442c89df493930f2012e; no voters:
I20260928 15:50:31.350214 24826 raft_consensus.cc:2781] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:31.350445 24826 raft_consensus.cc:686] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 LEADER]: Becoming Leader. State: Replica: 37eb8ffce9dd49d4901316d7d338cbf4, State: Running, Role: LEADER
I20260928 15:50:31.350776 24826 consensus_queue.cc:237] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } }
I20260928 15:50:31.352330 24851 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: TxnStatusTablet state changed. Reason: New leader 37eb8ffce9dd49d4901316d7d338cbf4. Latest consensus state: current_term: 1 leader_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.352645 24851 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:50:31.353449 24864 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:50:31.353734 24864 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:50:31.354051 24303 catalog_manager.cc:5495] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37eb8ffce9dd49d4901316d7d338cbf4 (127.22.157.194). New cstate: current_term: 1 leader_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } health_report { overall_health: UNKNOWN } } }
I20260928 15:50:31.382661 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.376551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:44523 (local address 127.0.0.1:43004)
0928 15:50:31.376800 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:31.376824 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:31.376978 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:31.377559 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:31.377564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:31.377588 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:50:31.377815 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.377825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.378944 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.378951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.379955 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.379968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.380768 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.380775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.380940 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.380998 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:50:31.382022 (+ 1024us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:50:31.382045 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:50:31.382132 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 15:50:31.382288 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 15:50:31.382665 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.376620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.22.157.194:44523)
0928 15:50:31.377193 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.377198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.377213 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:50:31.377288 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.377291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.377375 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.377535 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:31.377981 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.378810 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.380128 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.380580 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.380727 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.381206 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:31.381894 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:31.382059 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.382174 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.382254 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":254,"threads_started":1}
I20260928 15:50:31.387581 24464 raft_consensus.cc:1261] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Refusing update from remote peer 37eb8ffce9dd49d4901316d7d338cbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:31.387842 24748 raft_consensus.cc:1261] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e [term 1 FOLLOWER]: Refusing update from remote peer 37eb8ffce9dd49d4901316d7d338cbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:31.388463 24826 consensus_queue.cc:1035] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:31.388885 24851 consensus_queue.cc:1035] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:31.401598 24825 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: TxnStatusTablet state changed. Reason: New leader 37eb8ffce9dd49d4901316d7d338cbf4. Latest consensus state: current_term: 1 leader_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.402084 24825 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:31.404595 24868 mvcc.cc:205] Tried to move back new op lower bound from 7334341146156027904 to 7334341146014912512. Current Snapshot: MvccSnapshot[applied={T|T < 7334341146156027904}]
I20260928 15:50:31.406525 24826 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.406878 24826 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:50:31.408145 24828 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: TxnStatusTablet state changed. Reason: New leader 37eb8ffce9dd49d4901316d7d338cbf4. Latest consensus state: current_term: 1 leader_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.408468 24828 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:31.410058 24826 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.410414 24826 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P 37eb8ffce9dd49d4901316d7d338cbf4: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:50:31.413239 24870 mvcc.cc:205] Tried to move back new op lower bound from 7334341146156027904 to 7334341146014912512. Current Snapshot: MvccSnapshot[applied={T|T < 7334341146156027904}]
I20260928 15:50:31.414547 24825 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.414794 24825 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P 5efc3b7be87f4f5eaadc5b4129eda1a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:31.421675 24828 tablet_replica.cc:439] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 } } peers { permanent_uuid: "37eb8ffce9dd49d4901316d7d338cbf4" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44523 } } peers { permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 } } }
I20260928 15:50:31.421973 24828 tablet_replica.cc:442] T 79b6f55f777041da8e1d77b423bed8ab P c1cf98898121442c89df493930f2012e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:31.425814 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.418761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.22.157.194:44523)
0928 15:50:31.418941 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.418948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.419356 (+ 408us) server_negotiation.cc:407] Connection header received
0928 15:50:31.419592 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.419598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.419696 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.419811 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:31.420506 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.421675 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.423817 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.424503 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.424627 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.425064 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:31.425190 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:31.425280 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.425423 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.425503 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:50:31.442513 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.436669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:37183 (local address 127.22.157.194:44523)
0928 15:50:31.436829 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.436834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.437507 (+ 673us) server_negotiation.cc:407] Connection header received
0928 15:50:31.437702 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.437706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.437776 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.437884 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:31.438394 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.439182 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.440688 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.441166 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.441303 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.441719 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:31.441812 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:31.441904 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.442048 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.442117 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":33}
I20260928 15:50:31.442580 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.436790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:44523 (local address 127.22.157.194:37183)
0928 15:50:31.437372 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:31.437388 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:31.437558 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:31.437886 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:31.437890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:31.437901 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:50:31.438259 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.438268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.439313 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.439321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.440541 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:31.440556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.441369 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.441376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:31.441494 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.441546 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:50:31.441903 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:50:31.441914 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:50:31.442017 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:50:31.442145 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":4,"thread_start_us":314,"threads_started":1}
I20260928 15:50:31.452569 24464 raft_consensus.cc:1261] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Refusing update from remote peer 37eb8ffce9dd49d4901316d7d338cbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:31.452622 24747 raft_consensus.cc:1261] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [term 1 FOLLOWER]: Refusing update from remote peer 37eb8ffce9dd49d4901316d7d338cbf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:31.453176 24851 consensus_queue.cc:1035] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5efc3b7be87f4f5eaadc5b4129eda1a1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:31.453492 24826 consensus_queue.cc:1035] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1cf98898121442c89df493930f2012e" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:31.523715 24302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:31.523981 24302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:31.527551 24302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 552da18172a94b8b80bba489ab3aaad2: Sending DeleteTablet for 3 replicas of tablet 379bb0e741554ef78eaddd6af9cea7f6
I20260928 15:50:31.528453 24444 tablet_service.cc:1514] Processing DeleteTablet for tablet 379bb0e741554ef78eaddd6af9cea7f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:31 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:56368
I20260928 15:50:31.528591 24728 tablet_service.cc:1514] Processing DeleteTablet for tablet 379bb0e741554ef78eaddd6af9cea7f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:31 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:51854
I20260928 15:50:31.537086 24586 tablet_service.cc:1514] Processing DeleteTablet for tablet 379bb0e741554ef78eaddd6af9cea7f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:31 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:43002
I20260928 15:50:31.541383 24885 tablet_replica.cc:330] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: stopping tablet replica
I20260928 15:50:31.541741 24885 raft_consensus.cc:2229] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:31.542126 24885 raft_consensus.cc:2258] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:31.543407 24885 ts_tablet_manager.cc:1905] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:31.551328 24885 ts_tablet_manager.cc:1918] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:50:31.551496 24885 log.cc:1196] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/379bb0e741554ef78eaddd6af9cea7f6
I20260928 15:50:31.551900 24885 ts_tablet_manager.cc:1939] T 379bb0e741554ef78eaddd6af9cea7f6 P 5efc3b7be87f4f5eaadc5b4129eda1a1: Deleting consensus metadata
I20260928 15:50:31.553275 24888 tablet_replica.cc:330] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: stopping tablet replica
I20260928 15:50:31.553642 24888 raft_consensus.cc:2229] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:50:31.553884 24289 catalog_manager.cc:4841] TS 5efc3b7be87f4f5eaadc5b4129eda1a1 (127.22.157.193:37277): tablet 379bb0e741554ef78eaddd6af9cea7f6 (table test-table [id=e26ce3a982b74ad99dd728e14e5d4a93]) successfully deleted
I20260928 15:50:31.554704 24888 raft_consensus.cc:2258] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:31.555297 24886 tablet_replica.cc:330] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: stopping tablet replica
I20260928 15:50:31.555536 24886 raft_consensus.cc:2229] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:31.555795 24886 raft_consensus.cc:2258] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:31.556550 24888 ts_tablet_manager.cc:1905] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:31.556900 24886 ts_tablet_manager.cc:1905] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:31.564142 24886 ts_tablet_manager.cc:1918] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:50:31.564327 24886 log.cc:1196] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/379bb0e741554ef78eaddd6af9cea7f6
I20260928 15:50:31.564565 24888 ts_tablet_manager.cc:1918] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:50:31.564868 24886 ts_tablet_manager.cc:1939] T 379bb0e741554ef78eaddd6af9cea7f6 P c1cf98898121442c89df493930f2012e: Deleting consensus metadata
I20260928 15:50:31.564878 24888 log.cc:1196] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/379bb0e741554ef78eaddd6af9cea7f6
I20260928 15:50:31.565646 24888 ts_tablet_manager.cc:1939] T 379bb0e741554ef78eaddd6af9cea7f6 P 37eb8ffce9dd49d4901316d7d338cbf4: Deleting consensus metadata
I20260928 15:50:31.566368 24290 catalog_manager.cc:4841] TS c1cf98898121442c89df493930f2012e (127.22.157.195:36571): tablet 379bb0e741554ef78eaddd6af9cea7f6 (table test-table [id=e26ce3a982b74ad99dd728e14e5d4a93]) successfully deleted
I20260928 15:50:31.569033 24288 catalog_manager.cc:4841] TS 37eb8ffce9dd49d4901316d7d338cbf4 (127.22.157.194:44523): tablet 379bb0e741554ef78eaddd6af9cea7f6 (table test-table [id=e26ce3a982b74ad99dd728e14e5d4a93]) successfully deleted
Sep 28 15:50:31 dist-test-slave-1vw0 krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610630, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:50:31.586174 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.555412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.22.157.193:37277)
0928 15:50:31.555595 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.555603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.558156 (+ 2553us) server_negotiation.cc:407] Connection header received
0928 15:50:31.558650 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.558656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.558740 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.558812 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:31.561463 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.562655 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.565449 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.566163 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.566269 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.578826 (+ 12557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:31.578842 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:31.578856 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:31.578905 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:31.582274 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.583299 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.583307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.583310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.583459 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.584063 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.584068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.584072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.584487 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:31.584864 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.585575 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.585828 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 15:50:31.586963 24510 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56384
I20260928 15:50:31.604313 24858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.590714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.22.157.254:34751)
0928 15:50:31.590920 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.590926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.591326 (+ 400us) server_negotiation.cc:407] Connection header received
0928 15:50:31.591525 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.591531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.591624 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.591731 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:31.593124 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.593905 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.595325 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.595823 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.595970 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.598658 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:31.598668 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:31.598672 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:31.598752 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:31.601589 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.602281 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.602287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.602291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.602438 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.602816 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.602821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.602824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.603125 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:31.603387 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.603740 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.603973 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 15:50:31.605319 24302 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45876
I20260928 15:50:31.607683 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 24375
I20260928 15:50:31.622977 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.609049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56386 (local address 127.22.157.193:37277)
0928 15:50:31.609279 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.609287 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.610800 (+ 1513us) server_negotiation.cc:407] Connection header received
0928 15:50:31.611023 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.611029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.611108 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.611221 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:31.612367 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.613242 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.614500 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.614920 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.615055 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.617475 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:31.617480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:31.617483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:31.617520 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:31.620222 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.620970 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.620977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.620986 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.621126 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.621510 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.621513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.621515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.621801 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:31.622083 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.622400 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.622632 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 15:50:31.623633 24510 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56386
W20260928 15:50:31.624233 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 24375: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:31.624420 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 24375
I20260928 15:50:31.642580 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 24520
Sep 28 15:50:31 dist-test-slave-1vw0 krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610630, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:50:31.659832 24867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.643403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.22.157.194:44523)
0928 15:50:31.643605 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.643610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.643817 (+ 207us) server_negotiation.cc:407] Connection header received
0928 15:50:31.644068 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.644073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.644153 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.644281 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:31.645606 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.646497 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.647726 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.648237 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.648403 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.653084 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:31.653098 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:31.653117 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:31.653168 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:31.656392 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.657327 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.657344 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.657347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.657539 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.658041 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.658062 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.658065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.658515 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:31.658933 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.659217 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.659472 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 15:50:31.660559 24653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43024
W20260928 15:50:31.661357 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 24520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:31.661630 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 24520
I20260928 15:50:31.681684 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 24662
Sep 28 15:50:31 dist-test-slave-1vw0 krb5kdc[24257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610630, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:50:31.697468 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:31.682540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.22.157.195:36571)
0928 15:50:31.683085 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:31.683090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:31.683106 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:50:31.683164 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:31.683167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:31.683225 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:31.683327 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:31.684555 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.685347 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.686530 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:31.687061 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:31.687189 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:31.691666 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:31.691681 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:31.691698 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:31.691733 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:31.694675 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.695421 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.695429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.695431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.695586 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:31.695897 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:31.695900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:31.695902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:31.696272 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:31.696520 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:31.696884 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 15:50:31.697121 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":319,"threads_started":1}
W20260928 15:50:31.698177 24794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51872
W20260928 15:50:31.698864 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 24662: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:31.699002 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 24662
I20260928 15:50:31.720230 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 24270
I20260928 15:50:31.721408 24343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:31.844905 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 24270
2026-09-28T15:50:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4747 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:31 dist-test-slave-1vw0 krb5kdc[24897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:31 dist-test-slave-1vw0 krb5kdc[24897](info): set up 2 sockets
Sep 28 15:50:31 dist-test-slave-1vw0 krb5kdc[24897](info): commencing operation
krb5kdc: starting...
W20260928 15:50:32.937570 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:50:33 dist-test-slave-1vw0 krb5kdc[24897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:50:33.136667 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:44399
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:44399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:33.472638 24910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:33.473043 24910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:33.473138 24910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:33.480585 24910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:33.480708 24910 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:50:33.480831 24910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:33.480904 24910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:33.480957 24910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:33.481014 24910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:33.491492 24910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:44399
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:44399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:33.492910 24910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:33.494323 24910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:33.507078 24915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:33.507120 24916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:33.506963 24918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:33.508088 24910 server_base.cc:1034] running on GCE node
I20260928 15:50:33.509006 24910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:33.510483 24910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:33.511713 24910 hybrid_clock.cc:648] HybridClock initialized: now 1790610633511629 us; error 55 us; skew 500 ppm
I20260928 15:50:33.512023 24910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:33 dist-test-slave-1vw0 krb5kdc[24897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610633, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:33.515486 24910 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:50:33.517094 24910 webserver.cc:460] Webserver started at http://127.22.157.254:40203/ using document root <none> and password file <none>
I20260928 15:50:33.517851 24910 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:33.517973 24910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:33.518321 24910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:33.521514 24910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "088891443ba343ae971692d302adb6bd"
format_stamp: "Formatted at 2026-09-28 15:50:33 on dist-test-slave-1vw0"
server_key: "e0e6fe3eee7a71f8ba548e07834b892f"
server_key_iv: "0cd17cfbdb86225c8bb6495518492da7"
server_key_version: "encryptionkey@0"
I20260928 15:50:33.522339 24910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "088891443ba343ae971692d302adb6bd"
format_stamp: "Formatted at 2026-09-28 15:50:33 on dist-test-slave-1vw0"
server_key: "e0e6fe3eee7a71f8ba548e07834b892f"
server_key_iv: "0cd17cfbdb86225c8bb6495518492da7"
server_key_version: "encryptionkey@0"
I20260928 15:50:33.527957 24910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:50:33.532039 24925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:33.533604 24910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:33.533804 24910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "088891443ba343ae971692d302adb6bd"
format_stamp: "Formatted at 2026-09-28 15:50:33 on dist-test-slave-1vw0"
server_key: "e0e6fe3eee7a71f8ba548e07834b892f"
server_key_iv: "0cd17cfbdb86225c8bb6495518492da7"
server_key_version: "encryptionkey@0"
I20260928 15:50:33.534050 24910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:33.551450 24910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:33.556165 24910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:33.556459 24910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:33.579102 24910 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:44399
I20260928 15:50:33.579128 24986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:44399 every 8 connection(s)
I20260928 15:50:33.580567 24910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:33.584405 24987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:33.587455 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 24910
I20260928 15:50:33.587713 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:50:33.588512 23159 external_mini_cluster.cc:1428] Setting key caccd414c4505bd2907ea42da961a305
I20260928 15:50:33.597954 24987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd: Bootstrap starting.
I20260928 15:50:33.602185 24987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:33.603675 24987 log.cc:824] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:50:33 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:33.607097 24987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd: No bootstrap required, opened a new log
I20260928 15:50:33.613521 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:33.590881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.22.157.254:44399)
0928 15:50:33.591743 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:33.591756 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:33.591801 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:50:33.593622 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:33.593650 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:33.594087 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:33.594465 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:33.596245 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:33.597720 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:33.599551 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:33.600399 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:33.600539 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:33.605686 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:33.605695 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:33.605711 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:33.605747 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:33.609071 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:33.609993 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:33.609999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:33.610006 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:33.610182 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:33.610591 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:33.610597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:33.610599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:33.611142 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:33.611367 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:33.611855 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 15:50:33.612270 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":394,"threads_started":1}
I20260928 15:50:33.616902 24987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088891443ba343ae971692d302adb6bd" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44399 } }
I20260928 15:50:33.617311 24987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:33.617430 24987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088891443ba343ae971692d302adb6bd, State: Initialized, Role: FOLLOWER
I20260928 15:50:33.618040 24987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088891443ba343ae971692d302adb6bd" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44399 } }
I20260928 15:50:33.618311 24987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:33.618460 24987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:33.618633 24987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:33.621081 24987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088891443ba343ae971692d302adb6bd" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44399 } }
I20260928 15:50:33.621654 24987 leader_election.cc:302] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 088891443ba343ae971692d302adb6bd; no voters:
I20260928 15:50:33.622222 24987 leader_election.cc:288] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:33.622355 24993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:33.623086 24993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [term 1 LEADER]: Becoming Leader. State: Replica: 088891443ba343ae971692d302adb6bd, State: Running, Role: LEADER
I20260928 15:50:33.623595 24993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088891443ba343ae971692d302adb6bd" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44399 } }
I20260928 15:50:33.624320 24987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:33.627647 24995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 088891443ba343ae971692d302adb6bd. Latest consensus state: current_term: 1 leader_uuid: "088891443ba343ae971692d302adb6bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088891443ba343ae971692d302adb6bd" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44399 } } }
I20260928 15:50:33.628919 24995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:33.628652 24994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "088891443ba343ae971692d302adb6bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088891443ba343ae971692d302adb6bd" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44399 } } }
I20260928 15:50:33.629240 24994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:33.632879 25001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:33.637946 25001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:33.647152 25001 catalog_manager.cc:1347] Generated new cluster ID: 28671571c6584aa98bbac6e166811451
I20260928 15:50:33.647290 25001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:33.665088 25001 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:33.666453 25001 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:33.675611 25001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd: Generated new TSK 0
I20260928 15:50:33.676796 25001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:50:33.729614 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:44399
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:34.067788 25014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:34.068178 25014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:34.068385 25014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:34.076376 25014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:34.076581 25014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:34.076824 25014 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:50:34.076929 25014 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:50:34.077071 25014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:50:34.087790 25014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.157.254:44399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:34.089355 25014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:34.090919 25014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:34.105648 25019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:34.105703 25020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:34.105649 25022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:34.107175 25014 server_base.cc:1034] running on GCE node
I20260928 15:50:34.107946 25014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:34.109129 25014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:34.110397 25014 hybrid_clock.cc:648] HybridClock initialized: now 1790610634110360 us; error 57 us; skew 500 ppm
I20260928 15:50:34.110764 25014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:34 dist-test-slave-1vw0 krb5kdc[24897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610634, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:34.114341 25014 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:50:34.115880 25014 webserver.cc:460] Webserver started at http://127.22.157.193:33535/ using document root <none> and password file <none>
I20260928 15:50:34.116603 25014 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:34.116777 25014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:34.117120 25014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:34.120096 25014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "ff1f5754815c45f1a3545c76e803afa1"
format_stamp: "Formatted at 2026-09-28 15:50:34 on dist-test-slave-1vw0"
server_key: "faf4d20950d4dc793238f369d75409d6"
server_key_iv: "a61efd876203b6bd864c1914f328a56b"
server_key_version: "encryptionkey@0"
I20260928 15:50:34.120942 25014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "ff1f5754815c45f1a3545c76e803afa1"
format_stamp: "Formatted at 2026-09-28 15:50:34 on dist-test-slave-1vw0"
server_key: "faf4d20950d4dc793238f369d75409d6"
server_key_iv: "a61efd876203b6bd864c1914f328a56b"
server_key_version: "encryptionkey@0"
I20260928 15:50:34.126708 25014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:50:34.130729 25029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:34.132153 25014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 15:50:34.132359 25014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "ff1f5754815c45f1a3545c76e803afa1"
format_stamp: "Formatted at 2026-09-28 15:50:34 on dist-test-slave-1vw0"
server_key: "faf4d20950d4dc793238f369d75409d6"
server_key_iv: "a61efd876203b6bd864c1914f328a56b"
server_key_version: "encryptionkey@0"
I20260928 15:50:34.132589 25014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:34.156814 25014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:34.161559 25014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:34.161870 25014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:34.164614 25014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:34.164780 25014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:34.164929 25014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:34.165122 25014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:50:34 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610634, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:34.190361 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:34.165167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:58059 (local address 127.22.157.254:44399)
0928 15:50:34.165669 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:34.165676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:34.166919 (+ 1243us) server_negotiation.cc:407] Connection header received
0928 15:50:34.169931 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:34.169935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:34.170013 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:34.170167 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:34.172596 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.173962 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.176779 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.177403 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.177542 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:34.182451 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:34.182456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:34.182459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:34.182493 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:34.186099 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:34.187088 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:34.187092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:34.187094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:34.187208 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:34.187819 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:34.187825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:34.187830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:34.188285 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:34.188449 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:34.189437 (+ 988us) server_negotiation.cc:299] Negotiation successful
0928 15:50:34.189830 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":270,"threads_started":1}
I20260928 15:50:34.191094 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:34.165818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44399 (local address 127.22.157.193:58059)
0928 15:50:34.166722 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:34.166787 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:34.169588 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:34.170469 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:34.170484 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:34.171440 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:34.172357 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:34.172398 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.174156 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.174160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:34.176525 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:34.176547 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.177660 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.177668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:34.177929 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:34.178851 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:34.178876 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:34.182197 (+ 3321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:34.186270 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:34.186284 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:34.186303 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:34.186924 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:34.187333 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:34.187338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:34.187342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:34.187676 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:34.188453 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:34.188463 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:34.188960 (+ 497us) client_negotiation.cc:769] Sending connection context
0928 15:50:34.189321 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 15:50:34.189818 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":352,"threads_started":1}
I20260928 15:50:34.213423 25014 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:33469
I20260928 15:50:34.213449 25149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:33469 every 8 connection(s)
I20260928 15:50:34.214872 25014 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:50:34.215936 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:34.200258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:60331 (local address 127.22.157.254:44399)
0928 15:50:34.200474 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:34.200481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:34.201447 (+ 966us) server_negotiation.cc:407] Connection header received
0928 15:50:34.201670 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:34.201675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:34.201757 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:34.201863 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:34.203577 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.204503 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.206315 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.206891 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.207022 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:34.209765 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:34.209777 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:34.209780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:34.209816 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:34.213012 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:34.213931 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:34.213935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:34.213937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:34.214080 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:34.214454 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:34.214463 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:34.214465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:34.214756 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:34.214908 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:34.215346 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 15:50:34.215580 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:50:34.216116 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:34.200396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44399 (local address 127.22.157.193:60331)
0928 15:50:34.201319 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:34.201335 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:34.201512 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:34.201900 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:34.201905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:34.202720 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:34.203404 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:34.203427 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.204668 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.204677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:34.206086 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:34.206109 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.207295 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.207305 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:34.207569 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:34.208316 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:34.208336 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:34.209487 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:34.213159 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:34.213165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:34.213169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:34.213793 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:34.214200 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:34.214206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:34.214210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:34.214344 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:34.214878 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:34.214883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:34.215093 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 15:50:34.215232 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 15:50:34.215625 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":526,"threads_started":1}
I20260928 15:50:34.217458 25150 heartbeater.cc:347] Connected to a master server at 127.22.157.254:44399
I20260928 15:50:34.217830 25150 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:34.218886 25150 heartbeater.cc:511] Master 127.22.157.254:44399 requested a full tablet report, sending...
I20260928 15:50:34.221446 24942 ts_manager.cc:194] Registered new tserver with Master: ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469)
I20260928 15:50:34.222193 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 25014
I20260928 15:50:34.222389 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:50:34.223037 23159 external_mini_cluster.cc:1428] Setting key d0def8237afef6531812d943fd7e23fc
I20260928 15:50:34.223037 24942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:58059
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:50:34.267485 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:44399
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:34.592567 25154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:34.592952 25154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:34.593142 25154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:34.601336 25154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:34.601528 25154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:34.601774 25154 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:50:34.601850 25154 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:50:34.601923 25154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:50:34.613219 25154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.157.254:44399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:34.614568 25154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:34.615896 25154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:34.629451 25159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:34.629534 25160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:34.629493 25162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:34.630589 25154 server_base.cc:1034] running on GCE node
I20260928 15:50:34.631217 25154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:34.632130 25154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:34.633358 25154 hybrid_clock.cc:648] HybridClock initialized: now 1790610634633297 us; error 59 us; skew 500 ppm
I20260928 15:50:34.633699 25154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:34 dist-test-slave-1vw0 krb5kdc[24897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610634, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:34.637264 25154 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:50:34.638888 25154 webserver.cc:460] Webserver started at http://127.22.157.194:38469/ using document root <none> and password file <none>
I20260928 15:50:34.639595 25154 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:34.639748 25154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:34.640070 25154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:34.643066 25154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "aa34856ffc7d44e4932d9acb6451a0bd"
format_stamp: "Formatted at 2026-09-28 15:50:34 on dist-test-slave-1vw0"
server_key: "9ec8693d6f0ac1124691c955f875826e"
server_key_iv: "e95015096c9aa0a5e0b6f32e7bf313fb"
server_key_version: "encryptionkey@0"
I20260928 15:50:34.643828 25154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "aa34856ffc7d44e4932d9acb6451a0bd"
format_stamp: "Formatted at 2026-09-28 15:50:34 on dist-test-slave-1vw0"
server_key: "9ec8693d6f0ac1124691c955f875826e"
server_key_iv: "e95015096c9aa0a5e0b6f32e7bf313fb"
server_key_version: "encryptionkey@0"
I20260928 15:50:34.649087 25154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:50:34.652580 25169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:34.654084 25154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:34.654302 25154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "aa34856ffc7d44e4932d9acb6451a0bd"
format_stamp: "Formatted at 2026-09-28 15:50:34 on dist-test-slave-1vw0"
server_key: "9ec8693d6f0ac1124691c955f875826e"
server_key_iv: "e95015096c9aa0a5e0b6f32e7bf313fb"
server_key_version: "encryptionkey@0"
I20260928 15:50:34.654531 25154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:34.691495 25154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:34.695996 25154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:34.696307 25154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:34.698895 25154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:34.698997 25154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:34.699156 25154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:34.699250 25154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:50:34 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610634, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:34.724325 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:34.700069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:48611 (local address 127.22.157.254:44399)
0928 15:50:34.700295 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:34.700302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:34.701549 (+ 1247us) server_negotiation.cc:407] Connection header received
0928 15:50:34.703867 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:34.703872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:34.703968 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:34.704092 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:34.706850 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.707859 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.710190 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.711029 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.711181 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:34.716488 (+ 5307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:34.716494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:34.716499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:34.716579 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:34.720085 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:34.721425 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:34.721432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:34.721437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:34.721600 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:34.722123 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:34.722128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:34.722132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:34.722581 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:34.722755 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:34.723732 (+ 977us) server_negotiation.cc:299] Negotiation successful
0928 15:50:34.723958 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:50:34.726028 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:34.700467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44399 (local address 127.22.157.194:48611)
0928 15:50:34.701366 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:34.701421 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:34.703571 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:34.704414 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:34.704440 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:34.705469 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:34.706631 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:34.706664 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.708052 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.708062 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:34.709990 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:34.710044 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.711266 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.711272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:34.711450 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:34.712196 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:34.712243 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:34.716246 (+ 4003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:34.720237 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:34.720251 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:34.720273 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:34.721246 (+ 973us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:34.721728 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:34.721733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:34.721737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:34.721993 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:34.722735 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:34.722743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:34.723329 (+ 586us) client_negotiation.cc:769] Sending connection context
0928 15:50:34.723653 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 15:50:34.724220 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":404,"threads_started":1}
I20260928 15:50:34.744593 25291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:38869 every 8 connection(s)
I20260928 15:50:34.744593 25154 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:38869
I20260928 15:50:34.746368 25154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:50:34.748643 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:34.732966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:33273 (local address 127.22.157.254:44399)
0928 15:50:34.733584 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:34.733592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:34.734039 (+ 447us) server_negotiation.cc:407] Connection header received
0928 15:50:34.734378 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:34.734385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:34.734475 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:34.734557 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:34.736131 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.737238 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.739344 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.739867 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.740004 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:34.742296 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:34.742302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:34.742306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:34.742356 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:34.745696 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:34.746412 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:34.746429 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:34.746432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:34.746601 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:34.747005 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:34.747011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:34.747014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:34.747499 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:34.747660 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:34.748073 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 15:50:34.748283 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433}
I20260928 15:50:34.748759 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:34.733139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44399 (local address 127.22.157.194:33273)
0928 15:50:34.733915 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:34.733933 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:34.734149 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:34.734780 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:34.734787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:34.735421 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:34.735957 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:34.735968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.737447 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.737453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:34.739195 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:34.739209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:34.740078 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:34.740085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:34.740259 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:34.741040 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:34.741070 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:34.742117 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:34.745859 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:34.745863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:34.745866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:34.746283 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:34.746712 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:34.746716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:34.746718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:34.746834 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:34.747630 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:34.747635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:34.747821 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 15:50:34.747982 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 15:50:34.748300 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":484,"threads_started":1}
I20260928 15:50:34.748982 25292 heartbeater.cc:347] Connected to a master server at 127.22.157.254:44399
I20260928 15:50:34.749332 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 25154
I20260928 15:50:34.749459 25292 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:34.749514 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:50:34.750116 23159 external_mini_cluster.cc:1428] Setting key b4e243174520eb386cbbe37fd25fa844
I20260928 15:50:34.750300 25292 heartbeater.cc:511] Master 127.22.157.254:44399 requested a full tablet report, sending...
I20260928 15:50:34.751854 24941 ts_manager.cc:194] Registered new tserver with Master: aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194:38869)
I20260928 15:50:34.753170 24941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:48611
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:50:34.792549 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:44399
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:35.117030 25296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:35.117307 25296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:35.117409 25296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:35.124715 25296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:35.124946 25296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:35.125065 25296 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 15:50:35.125123 25296 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:50:35.125192 25296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:50:35.135365 25296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.157.254:44399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:35.136763 25296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:35.138247 25296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:35.151605 25302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:35.151685 25301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:35.151969 25304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:35.153007 25296 server_base.cc:1034] running on GCE node
I20260928 15:50:35.153884 25296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:35.155020 25296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:35.156250 25296 hybrid_clock.cc:648] HybridClock initialized: now 1790610635156199 us; error 36 us; skew 500 ppm
I20260928 15:50:35.156690 25296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:35 dist-test-slave-1vw0 krb5kdc[24897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610635, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:35.160259 25296 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:50:35.162047 25296 webserver.cc:460] Webserver started at http://127.22.157.195:33517/ using document root <none> and password file <none>
I20260928 15:50:35.162743 25296 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:35.162846 25296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:35.163116 25296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:35.166229 25296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "ed7292110a4e46f7980e0adbc7967bcf"
format_stamp: "Formatted at 2026-09-28 15:50:35 on dist-test-slave-1vw0"
server_key: "537962745b31813ba2179dbf8ed40670"
server_key_iv: "316e4c4ac85545f9d75baef158d61a38"
server_key_version: "encryptionkey@0"
I20260928 15:50:35.166970 25296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "ed7292110a4e46f7980e0adbc7967bcf"
format_stamp: "Formatted at 2026-09-28 15:50:35 on dist-test-slave-1vw0"
server_key: "537962745b31813ba2179dbf8ed40670"
server_key_iv: "316e4c4ac85545f9d75baef158d61a38"
server_key_version: "encryptionkey@0"
I20260928 15:50:35.172420 25296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 15:50:35.176184 25311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:35.177779 25296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:35.177960 25296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "ed7292110a4e46f7980e0adbc7967bcf"
format_stamp: "Formatted at 2026-09-28 15:50:35 on dist-test-slave-1vw0"
server_key: "537962745b31813ba2179dbf8ed40670"
server_key_iv: "316e4c4ac85545f9d75baef158d61a38"
server_key_version: "encryptionkey@0"
I20260928 15:50:35.178194 25296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:35.195063 25296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:35.199937 25296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:35.200311 25296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:35.203420 25296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:35.203585 25296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:35.203724 25296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:35.203850 25296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:50:35 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610635, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:35.226423 25150 heartbeater.cc:503] Master 127.22.157.254:44399 was elected leader, sending a full tablet report...
I20260928 15:50:35.234382 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.204178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:41501 (local address 127.22.157.254:44399)
0928 15:50:35.204399 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.204406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.206046 (+ 1640us) server_negotiation.cc:407] Connection header received
0928 15:50:35.209458 (+ 3412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.209469 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.209579 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.209683 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:35.213353 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.214803 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.217801 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.218494 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.218643 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.224935 (+ 6292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:35.224961 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:35.224968 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:35.225029 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:35.229646 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:35.230713 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:35.230722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:35.230728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:35.230940 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:35.231563 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:35.231571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:35.231577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:35.232106 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:35.232288 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.233516 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.233874 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 15:50:35.235515 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.204847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44399 (local address 127.22.157.195:41501)
0928 15:50:35.205810 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:35.205881 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:35.208942 (+ 3061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:35.210413 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:35.210430 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:35.211550 (+ 1120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:35.213003 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.213059 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.214990 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.214996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.217532 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.217558 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.219166 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.219178 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.219500 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.220463 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:35.220511 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:35.224542 (+ 4031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:35.229886 (+ 5344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:35.229895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:35.229911 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:35.230547 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:35.231108 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:35.231113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:35.231115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:35.231370 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:35.232294 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:35.232307 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:35.232972 (+ 665us) client_negotiation.cc:769] Sending connection context
0928 15:50:35.233273 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 15:50:35.233746 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":362,"threads_started":1}
I20260928 15:50:35.259637 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.242099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:44737 (local address 127.22.157.254:44399)
0928 15:50:35.242298 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.242304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.243231 (+ 927us) server_negotiation.cc:407] Connection header received
0928 15:50:35.243443 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.243448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.243543 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.243662 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:35.244648 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.245664 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.248438 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.248984 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.249199 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.251924 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:35.251929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:35.251933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:35.251968 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:35.256268 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:35.257226 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:35.257233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:35.257241 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:35.257400 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:35.257826 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:35.257833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:35.257836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:35.258137 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:35.258304 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.259030 (+ 726us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.259278 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:50:35.259712 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.242177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44399 (local address 127.22.157.195:44737)
0928 15:50:35.243112 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:35.243128 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:35.243307 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:35.243688 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:35.243693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:35.244184 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:35.244483 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.244492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.245871 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.245880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.248229 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.248257 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.249502 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.249511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.249675 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.250686 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:35.250715 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:35.251696 (+ 981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:35.256492 (+ 4796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:35.256496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:35.256499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:35.257063 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:35.257580 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:35.257584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:35.257586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:35.257713 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:35.258269 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:35.258274 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:35.258532 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 15:50:35.258716 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 15:50:35.259155 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":641,"threads_started":1}
I20260928 15:50:35.272907 25296 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:46251
I20260928 15:50:35.273006 25431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:46251 every 8 connection(s)
I20260928 15:50:35.274305 25296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:50:35.276594 25432 heartbeater.cc:347] Connected to a master server at 127.22.157.254:44399
I20260928 15:50:35.276881 25432 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:35.277601 25432 heartbeater.cc:511] Master 127.22.157.254:44399 requested a full tablet report, sending...
I20260928 15:50:35.279074 24941 ts_manager.cc:194] Registered new tserver with Master: ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251)
I20260928 15:50:35.280019 24941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:41501
I20260928 15:50:35.283384 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 25296
I20260928 15:50:35.283538 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:50:35.283944 23159 external_mini_cluster.cc:1428] Setting key 7953485e711bab11883db795a4fe2c5a
I20260928 15:50:35.286952 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:50:35 dist-test-slave-1vw0 krb5kdc[24897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:50:35 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:35.338958 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.323337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.22.157.254:44399)
0928 15:50:35.323523 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.323528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.323824 (+ 296us) server_negotiation.cc:407] Connection header received
0928 15:50:35.323973 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.323978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.324050 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.324188 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:35.325837 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.326625 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.327931 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.328450 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.328548 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.332672 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:35.332677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:35.332681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:35.332785 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:35.335670 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:35.336411 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:35.336416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:35.336418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:35.336561 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:35.337077 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:35.337082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:35.337084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:35.337454 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:35.337833 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.338191 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.338462 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:50:35.343415 24941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:50:35.350382 24941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:50:35.379793 25446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.363991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:46251 (local address 127.0.0.1:59658)
0928 15:50:35.365135 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:35.365199 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:35.365418 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:35.368510 (+ 3092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:35.368522 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:35.368581 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:35.369139 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.369164 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.372307 (+ 3143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.372320 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.375726 (+ 3406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.375753 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.378649 (+ 2896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.378657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.378877 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.378937 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 15:50:35.379075 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 15:50:35.379333 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":13771,"threads_started":1}
I20260928 15:50:35.380379 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.364492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.22.157.195:46251)
0928 15:50:35.366953 (+ 2461us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.366966 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.367010 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:50:35.367135 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.367147 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.367480 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.369391 (+ 1911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:35.369827 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.372066 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.376026 (+ 3960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.378026 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.378233 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.378427 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.379837 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.380014 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2248,"thread_start_us":2039,"threads_started":1}
I20260928 15:50:35.384569 25362 tablet_service.cc:1467] Processing CreateTablet for tablet 422f668ee87e49e5bcb7e6eb9c6cbf21 (DEFAULT_TABLE table=test-table [id=8d17ee38f82443ed9cc1fddce51deef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:35.386175 25362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422f668ee87e49e5bcb7e6eb9c6cbf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:35.386677 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.364223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:33469 (local address 127.0.0.1:55108)
0928 15:50:35.377558 (+ 13335us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:35.377579 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:35.377700 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:35.379162 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:35.379167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:35.379196 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:35.379624 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.379636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.381487 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.381493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.383817 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.383833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.385740 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.385748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.385990 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.386026 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:50:35.386126 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:50:35.386285 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13204,"mutex_wait_us":50,"thread_start_us":391,"threads_started":1}
I20260928 15:50:35.386678 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.364466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.22.157.193:33469)
0928 15:50:35.378443 (+ 13977us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.378454 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.378487 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:50:35.378603 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.378616 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.378905 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.379844 (+ 939us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:35.380083 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.381346 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.383975 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.385495 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.385672 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.385843 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.386204 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.386369 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13807,"thread_start_us":13674,"threads_started":1}
I20260928 15:50:35.389254 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.366590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:38869 (local address 127.0.0.1:54918)
0928 15:50:35.379398 (+ 12808us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:35.379414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:35.379576 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:35.380502 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:35.380510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:35.380539 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:35.382197 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.382210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.383889 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.383894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.385959 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.385972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.388032 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.388039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.388576 (+ 537us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.388620 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 15:50:35.388702 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:50:35.388882 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12669,"mutex_wait_us":1,"thread_start_us":12504,"threads_started":1}
I20260928 15:50:35.389464 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.367275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.22.157.194:38869)
0928 15:50:35.378901 (+ 11626us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.378913 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.379534 (+ 621us) server_negotiation.cc:407] Connection header received
0928 15:50:35.379778 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.379786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.380045 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.380264 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:35.382444 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.383742 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.386129 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.387504 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.387694 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.388821 (+ 1127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.388918 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.389101 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11426,"thread_start_us":11282,"threads_started":1}
I20260928 15:50:35.390803 25080 tablet_service.cc:1467] Processing CreateTablet for tablet 422f668ee87e49e5bcb7e6eb9c6cbf21 (DEFAULT_TABLE table=test-table [id=8d17ee38f82443ed9cc1fddce51deef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:35.392033 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422f668ee87e49e5bcb7e6eb9c6cbf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:35.393220 25221 tablet_service.cc:1467] Processing CreateTablet for tablet 422f668ee87e49e5bcb7e6eb9c6cbf21 (DEFAULT_TABLE table=test-table [id=8d17ee38f82443ed9cc1fddce51deef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:35.394670 25221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422f668ee87e49e5bcb7e6eb9c6cbf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:35.401911 25451 tablet_bootstrap.cc:492] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: Bootstrap starting.
I20260928 15:50:35.405849 25452 tablet_bootstrap.cc:492] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: Bootstrap starting.
I20260928 15:50:35.406157 25451 tablet_bootstrap.cc:654] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:35.407431 25451 log.cc:824] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:35.409582 25452 tablet_bootstrap.cc:654] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:35.410226 25454 tablet_bootstrap.cc:492] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: Bootstrap starting.
I20260928 15:50:35.410984 25452 log.cc:824] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:35.411060 25451 tablet_bootstrap.cc:492] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: No bootstrap required, opened a new log
I20260928 15:50:35.411348 25451 ts_tablet_manager.cc:1397] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 15:50:35.413694 25452 tablet_bootstrap.cc:492] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: No bootstrap required, opened a new log
I20260928 15:50:35.413972 25454 tablet_bootstrap.cc:654] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:35.413992 25452 ts_tablet_manager.cc:1397] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 15:50:35.415297 25454 log.cc:824] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:35.422500 25454 tablet_bootstrap.cc:492] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: No bootstrap required, opened a new log
I20260928 15:50:35.422434 25452 raft_consensus.cc:348] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } }
I20260928 15:50:35.422840 25454 ts_tablet_manager.cc:1397] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260928 15:50:35.422922 25452 raft_consensus.cc:374] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:35.423053 25452 raft_consensus.cc:729] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1f5754815c45f1a3545c76e803afa1, State: Initialized, Role: FOLLOWER
I20260928 15:50:35.423207 25451 raft_consensus.cc:348] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } }
I20260928 15:50:35.423682 25451 raft_consensus.cc:374] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:35.423799 25451 raft_consensus.cc:729] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7292110a4e46f7980e0adbc7967bcf, State: Initialized, Role: FOLLOWER
I20260928 15:50:35.423758 25452 consensus_queue.cc:260] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } }
I20260928 15:50:35.424952 25451 consensus_queue.cc:260] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } }
I20260928 15:50:35.426321 25432 heartbeater.cc:503] Master 127.22.157.254:44399 was elected leader, sending a full tablet report...
I20260928 15:50:35.426846 25451 ts_tablet_manager.cc:1428] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 15:50:35.427887 25452 ts_tablet_manager.cc:1428] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260928 15:50:35.428716 25457 raft_consensus.cc:482] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:35.429029 25457 raft_consensus.cc:504] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } }
I20260928 15:50:35.430742 25457 leader_election.cc:288] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251), aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194:38869)
I20260928 15:50:35.435197 25454 raft_consensus.cc:348] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } }
I20260928 15:50:35.435796 25454 raft_consensus.cc:374] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:35.435959 25454 raft_consensus.cc:729] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa34856ffc7d44e4932d9acb6451a0bd, State: Initialized, Role: FOLLOWER
I20260928 15:50:35.436797 25454 consensus_queue.cc:260] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } }
I20260928 15:50:35.438129 25292 heartbeater.cc:503] Master 127.22.157.254:44399 was elected leader, sending a full tablet report...
I20260928 15:50:35.439057 25454 ts_tablet_manager.cc:1428] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: Time spent starting tablet: real 0.016s user 0.006s sys 0.007s
I20260928 15:50:35.443954 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.431750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:47291 (local address 127.22.157.194:38869)
0928 15:50:35.431954 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.431960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.435628 (+ 3668us) server_negotiation.cc:407] Connection header received
0928 15:50:35.435868 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.435877 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.435991 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.436102 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:35.437311 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.438597 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.441800 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.442972 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.443164 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.443241 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.443550 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.443688 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:50:35.444058 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.431126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:38869 (local address 127.22.157.193:47291)
0928 15:50:35.435498 (+ 4372us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:35.435527 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:35.435691 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:35.436370 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:35.436378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:35.436419 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:35.437073 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.437097 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.439416 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.439421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.441636 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.441649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.443192 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.443200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.443381 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.443420 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:50:35.443512 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:50:35.443657 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4241,"mutex_wait_us":65,"thread_start_us":4075,"threads_started":1}
I20260928 15:50:35.444309 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.432108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:60799 (local address 127.22.157.195:46251)
0928 15:50:35.432306 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.432313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.437556 (+ 5243us) server_negotiation.cc:407] Connection header received
0928 15:50:35.437849 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.437854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.437975 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.438058 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:35.439147 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.440096 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.442394 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.443275 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.443435 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.443523 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.443882 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.444026 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:50:35.444398 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.431198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:46251 (local address 127.22.157.193:60799)
0928 15:50:35.437415 (+ 6217us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:35.437440 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:35.437688 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:35.438432 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:35.438441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:35.438472 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:35.438959 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.438974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.440236 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.440241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.442246 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.442258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.443513 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.443521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.443702 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.443734 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:50:35.443836 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:50:35.443971 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5976,"mutex_wait_us":48,"thread_start_us":5781,"threads_started":1}
I20260928 15:50:35.445307 25241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422f668ee87e49e5bcb7e6eb9c6cbf21" candidate_uuid: "ff1f5754815c45f1a3545c76e803afa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" is_pre_election: true
I20260928 15:50:35.445538 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422f668ee87e49e5bcb7e6eb9c6cbf21" candidate_uuid: "ff1f5754815c45f1a3545c76e803afa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf" is_pre_election: true
I20260928 15:50:35.445914 25241 raft_consensus.cc:2454] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1f5754815c45f1a3545c76e803afa1 in term 0.
I20260928 15:50:35.446019 25382 raft_consensus.cc:2454] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1f5754815c45f1a3545c76e803afa1 in term 0.
I20260928 15:50:35.446712 25030 leader_election.cc:302] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ff1f5754815c45f1a3545c76e803afa1; no voters:
I20260928 15:50:35.447192 25457 raft_consensus.cc:2781] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:35.447350 25457 raft_consensus.cc:482] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:35.447466 25457 raft_consensus.cc:3037] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:35.449816 25457 raft_consensus.cc:504] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } }
I20260928 15:50:35.450439 25457 leader_election.cc:288] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [CANDIDATE]: Term 1 election: Requested vote from peers ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251), aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194:38869)
I20260928 15:50:35.450969 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422f668ee87e49e5bcb7e6eb9c6cbf21" candidate_uuid: "ff1f5754815c45f1a3545c76e803afa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf"
I20260928 15:50:35.450970 25241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422f668ee87e49e5bcb7e6eb9c6cbf21" candidate_uuid: "ff1f5754815c45f1a3545c76e803afa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa34856ffc7d44e4932d9acb6451a0bd"
I20260928 15:50:35.451191 25241 raft_consensus.cc:3037] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:35.451191 25382 raft_consensus.cc:3037] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:35.454583 25382 raft_consensus.cc:2454] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1f5754815c45f1a3545c76e803afa1 in term 1.
I20260928 15:50:35.454583 25241 raft_consensus.cc:2454] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1f5754815c45f1a3545c76e803afa1 in term 1.
I20260928 15:50:35.455088 25030 leader_election.cc:302] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ed7292110a4e46f7980e0adbc7967bcf, ff1f5754815c45f1a3545c76e803afa1; no voters:
I20260928 15:50:35.455425 25457 raft_consensus.cc:2781] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:35.455924 25457 raft_consensus.cc:686] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 1 LEADER]: Becoming Leader. State: Replica: ff1f5754815c45f1a3545c76e803afa1, State: Running, Role: LEADER
I20260928 15:50:35.456424 25457 consensus_queue.cc:237] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } }
I20260928 15:50:35.461593 24942 catalog_manager.cc:5495] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193). New cstate: current_term: 1 leader_uuid: "ff1f5754815c45f1a3545c76e803afa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } health_report { overall_health: UNKNOWN } } }
W20260928 15:50:35.472556 25151 tablet.cc:2369] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:50:35.498414 25293 tablet.cc:2369] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 15:50:35 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610633, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:35.512893 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.499285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44399 (local address 127.0.0.1:60894)
0928 15:50:35.499905 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:35.499920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:35.500101 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:35.500433 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:35.500437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:35.501251 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:35.501559 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.501570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.502855 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.502861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.504018 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.504030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.505034 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.505038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.505154 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.505751 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:35.505772 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:35.507915 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:35.510716 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:35.510726 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:35.510732 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:35.511096 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:35.511411 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:35.511414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:35.511416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:35.511553 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:35.512015 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:35.512020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:35.512161 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 15:50:35.512288 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:50:35.512491 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":343,"threads_started":1}
I20260928 15:50:35.513016 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.499285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.22.157.254:44399)
0928 15:50:35.499458 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.499466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.500018 (+ 552us) server_negotiation.cc:407] Connection header received
0928 15:50:35.500229 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.500239 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.500306 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.500408 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:35.501716 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.502699 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.504158 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.504860 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.504964 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.508084 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:35.508088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:35.508091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:35.508119 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:35.510608 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:35.511192 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:35.511196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:35.511198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:35.511320 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:35.511648 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:35.511651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:35.511653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:35.511913 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:35.512021 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.512377 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.512635 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":60}
I20260928 15:50:35.517902 24942 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:60894:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:50:35.518868 24942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260928 15:50:35.525995 25433 tablet.cc:2369] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:35.525704 25080 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba731898d374e9791332f1055db2b66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4d39a5f19324998a41059b75dce157a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:50:35.525895 25221 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba731898d374e9791332f1055db2b66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4d39a5f19324998a41059b75dce157a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:50:35.526193 25362 tablet_service.cc:1467] Processing CreateTablet for tablet 3ba731898d374e9791332f1055db2b66 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b4d39a5f19324998a41059b75dce157a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:50:35.526787 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba731898d374e9791332f1055db2b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:35.526836 25221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba731898d374e9791332f1055db2b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:35.526993 25362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ba731898d374e9791332f1055db2b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:35.532064 25452 tablet_bootstrap.cc:492] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: Bootstrap starting.
I20260928 15:50:35.533645 25452 tablet_bootstrap.cc:654] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:35.534229 25454 tablet_bootstrap.cc:492] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: Bootstrap starting.
I20260928 15:50:35.534333 25451 tablet_bootstrap.cc:492] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: Bootstrap starting.
I20260928 15:50:35.535964 25454 tablet_bootstrap.cc:654] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:35.536121 25451 tablet_bootstrap.cc:654] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:35.536257 25452 tablet_bootstrap.cc:492] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: No bootstrap required, opened a new log
I20260928 15:50:35.536413 25452 ts_tablet_manager.cc:1397] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 15:50:35.537376 25452 raft_consensus.cc:348] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:35.537779 25452 raft_consensus.cc:374] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:35.537931 25452 raft_consensus.cc:729] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1f5754815c45f1a3545c76e803afa1, State: Initialized, Role: FOLLOWER
I20260928 15:50:35.538313 25452 consensus_queue.cc:260] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:35.538575 25454 tablet_bootstrap.cc:492] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: No bootstrap required, opened a new log
I20260928 15:50:35.538718 25454 ts_tablet_manager.cc:1397] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 15:50:35.538815 25457 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.539069 25457 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:35.539145 25451 tablet_bootstrap.cc:492] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: No bootstrap required, opened a new log
I20260928 15:50:35.539271 25451 ts_tablet_manager.cc:1397] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 15:50:35.539507 25454 raft_consensus.cc:348] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:35.539728 25454 raft_consensus.cc:374] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:35.539814 25454 raft_consensus.cc:729] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa34856ffc7d44e4932d9acb6451a0bd, State: Initialized, Role: FOLLOWER
I20260928 15:50:35.540141 25452 ts_tablet_manager.cc:1428] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 15:50:35.540006 25451 raft_consensus.cc:348] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:35.540047 25454 consensus_queue.cc:260] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:35.540233 25451 raft_consensus.cc:374] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:35.540323 25451 raft_consensus.cc:729] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7292110a4e46f7980e0adbc7967bcf, State: Initialized, Role: FOLLOWER
I20260928 15:50:35.540747 25454 ts_tablet_manager.cc:1428] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:50:35.540617 25451 consensus_queue.cc:260] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:35.540525 25461 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.540982 25461 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:35.541086 25458 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.541393 25458 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:35.542091 25451 ts_tablet_manager.cc:1428] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:50:35.882031 25457 raft_consensus.cc:482] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:35.882272 25457 raft_consensus.cc:504] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:35.883056 25457 leader_election.cc:288] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194:38869), ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251)
I20260928 15:50:35.883688 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba731898d374e9791332f1055db2b66" candidate_uuid: "ff1f5754815c45f1a3545c76e803afa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf" is_pre_election: true
I20260928 15:50:35.883688 25241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba731898d374e9791332f1055db2b66" candidate_uuid: "ff1f5754815c45f1a3545c76e803afa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" is_pre_election: true
I20260928 15:50:35.883955 25382 raft_consensus.cc:2454] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1f5754815c45f1a3545c76e803afa1 in term 0.
I20260928 15:50:35.883997 25241 raft_consensus.cc:2454] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff1f5754815c45f1a3545c76e803afa1 in term 0.
I20260928 15:50:35.884425 25030 leader_election.cc:302] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ff1f5754815c45f1a3545c76e803afa1; no voters:
I20260928 15:50:35.884704 25457 raft_consensus.cc:2781] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:35.884833 25457 raft_consensus.cc:482] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:35.884935 25457 raft_consensus.cc:3037] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:35.886940 25457 raft_consensus.cc:504] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:35.887601 25457 leader_election.cc:288] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [CANDIDATE]: Term 1 election: Requested vote from peers aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194:38869), ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251)
I20260928 15:50:35.888031 25241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba731898d374e9791332f1055db2b66" candidate_uuid: "ff1f5754815c45f1a3545c76e803afa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa34856ffc7d44e4932d9acb6451a0bd"
I20260928 15:50:35.888068 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ba731898d374e9791332f1055db2b66" candidate_uuid: "ff1f5754815c45f1a3545c76e803afa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf"
I20260928 15:50:35.888276 25241 raft_consensus.cc:3037] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:35.888276 25382 raft_consensus.cc:3037] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:35.891284 25382 raft_consensus.cc:2454] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1f5754815c45f1a3545c76e803afa1 in term 1.
I20260928 15:50:35.891283 25241 raft_consensus.cc:2454] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1f5754815c45f1a3545c76e803afa1 in term 1.
I20260928 15:50:35.891810 25030 leader_election.cc:302] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ed7292110a4e46f7980e0adbc7967bcf, ff1f5754815c45f1a3545c76e803afa1; no voters:
I20260928 15:50:35.892124 25457 raft_consensus.cc:2781] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:35.892325 25457 raft_consensus.cc:686] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [term 1 LEADER]: Becoming Leader. State: Replica: ff1f5754815c45f1a3545c76e803afa1, State: Running, Role: LEADER
I20260928 15:50:35.892586 25457 consensus_queue.cc:237] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:35.894467 25462 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: TxnStatusTablet state changed. Reason: New leader ff1f5754815c45f1a3545c76e803afa1. Latest consensus state: current_term: 1 leader_uuid: "ff1f5754815c45f1a3545c76e803afa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.894794 25462 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:50:35.895895 24942 catalog_manager.cc:5495] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193). New cstate: current_term: 1 leader_uuid: "ff1f5754815c45f1a3545c76e803afa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } health_report { overall_health: UNKNOWN } } }
I20260928 15:50:35.904588 25474 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:50:35.904896 25474 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:50:35.916812 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.909927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:33469 (local address 127.0.0.1:55116)
0928 15:50:35.910176 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:35.910201 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:35.910389 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:35.911103 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:35.911107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:35.911118 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:50:35.911276 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.911283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.912925 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.912930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.913975 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:35.913988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.915003 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.915008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:35.915156 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.915211 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:50:35.916272 (+ 1061us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:50:35.916296 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:50:35.916381 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 15:50:35.916502 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:50:35.916920 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.910098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.22.157.193:33469)
0928 15:50:35.910663 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.910669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.910695 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:50:35.910786 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.910792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.910889 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.911055 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:35.911432 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.912794 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.914125 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.914838 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.914968 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.915426 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:35.916147 (+ 721us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:35.916293 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.916418 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.916502 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":248,"threads_started":1}
I20260928 15:50:35.923326 25462 consensus_queue.cc:1035] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:35.925572 25241 raft_consensus.cc:1261] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Refusing update from remote peer ff1f5754815c45f1a3545c76e803afa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:35.925905 25381 raft_consensus.cc:1261] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Refusing update from remote peer ff1f5754815c45f1a3545c76e803afa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:35.926622 25480 consensus_queue.cc:1035] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:35.927121 25462 consensus_queue.cc:1035] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:35.949568 25458 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: TxnStatusTablet state changed. Reason: New leader ff1f5754815c45f1a3545c76e803afa1. Latest consensus state: current_term: 1 leader_uuid: "ff1f5754815c45f1a3545c76e803afa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.949883 25458 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:35.950939 25457 consensus_queue.cc:1035] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:35.953894 25479 mvcc.cc:205] Tried to move back new op lower bound from 7334341164740849664 to 7334341164618969088. Current Snapshot: MvccSnapshot[applied={T|T < 7334341164740849664}]
I20260928 15:50:35.956202 25461 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: TxnStatusTablet state changed. Reason: New leader ff1f5754815c45f1a3545c76e803afa1. Latest consensus state: current_term: 1 leader_uuid: "ff1f5754815c45f1a3545c76e803afa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.956485 25461 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:35.958706 25480 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff1f5754815c45f1a3545c76e803afa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.958993 25480 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:50:35.964783 25458 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff1f5754815c45f1a3545c76e803afa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.965099 25458 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P ed7292110a4e46f7980e0adbc7967bcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:35.971237 25457 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff1f5754815c45f1a3545c76e803afa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.971652 25457 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P ff1f5754815c45f1a3545c76e803afa1: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:50:35.972404 25482 mvcc.cc:205] Tried to move back new op lower bound from 7334341164740849664 to 7334341164618969088. Current Snapshot: MvccSnapshot[applied={T|T < 7334341164740849664}]
I20260928 15:50:35.977804 25486 tablet_replica.cc:439] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff1f5754815c45f1a3545c76e803afa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } } }
I20260928 15:50:35.978483 25486 tablet_replica.cc:442] T 3ba731898d374e9791332f1055db2b66 P aa34856ffc7d44e4932d9acb6451a0bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:50:35.988147 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.970035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.22.157.193:33469)
0928 15:50:35.970211 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.970218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:35.981654 (+ 11436us) server_negotiation.cc:407] Connection header received
0928 15:50:35.981884 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:35.981890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:35.981992 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:35.982098 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:35.983062 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.984196 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.985921 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:35.986613 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:35.986757 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:35.987282 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:35.987419 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:35.987495 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:35.987754 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 15:50:35.987829 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 15:50:36.006946 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.999131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:33147 (local address 127.22.157.193:33469)
0928 15:50:35.999352 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:35.999359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:36.001042 (+ 1683us) server_negotiation.cc:407] Connection header received
0928 15:50:36.001213 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:36.001218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:36.001327 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:36.001433 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:36.001862 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.003071 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.004648 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.005349 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.005512 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.006054 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:36.006181 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:36.006280 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:36.006437 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 15:50:36.006522 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:50:36.006964 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:35.999030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:33469 (local address 127.22.157.193:33147)
0928 15:50:36.000940 (+ 1910us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:36.000956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:36.001087 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:36.001444 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:36.001448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:36.001456 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:50:36.001707 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.001715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.003225 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.003229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.004512 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.004526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.005595 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.005600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.005763 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.005833 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:50:36.006314 (+ 481us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:50:36.006324 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:50:36.006397 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 15:50:36.006502 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1751,"mutex_wait_us":3,"thread_start_us":1547,"threads_started":1}
I20260928 15:50:36.073998 24941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:36.074293 24941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:36.077571 24941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd: Sending DeleteTablet for 3 replicas of tablet 422f668ee87e49e5bcb7e6eb9c6cbf21
I20260928 15:50:36.078883 25080 tablet_service.cc:1514] Processing DeleteTablet for tablet 422f668ee87e49e5bcb7e6eb9c6cbf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:36 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:55108
I20260928 15:50:36.078894 25362 tablet_service.cc:1514] Processing DeleteTablet for tablet 422f668ee87e49e5bcb7e6eb9c6cbf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:36 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:59658
I20260928 15:50:36.078918 25221 tablet_service.cc:1514] Processing DeleteTablet for tablet 422f668ee87e49e5bcb7e6eb9c6cbf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:36 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:54918
I20260928 15:50:36.080904 24941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:50:36.082088 24941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:50:36.085193 25500 tablet_replica.cc:330] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: stopping tablet replica
I20260928 15:50:36.085410 25499 tablet_replica.cc:330] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: stopping tablet replica
I20260928 15:50:36.085741 25500 raft_consensus.cc:2229] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:36.085829 25499 raft_consensus.cc:2229] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:50:36.086138 25500 raft_consensus.cc:2258] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:36.086539 25499 raft_consensus.cc:2258] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:36.087208 25498 tablet_replica.cc:330] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: stopping tablet replica
I20260928 15:50:36.088619 25500 ts_tablet_manager.cc:1905] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:36.088639 25499 ts_tablet_manager.cc:1905] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:36.092540 25221 tablet_service.cc:1467] Processing CreateTablet for tablet 45de63e3c7cf4efbabe2f76d52c14f50 (DEFAULT_TABLE table=test-table [id=80417385beb24119a8fd0bd0bb9d2324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:36.093007 25221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45de63e3c7cf4efbabe2f76d52c14f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:36.093744 25080 tablet_service.cc:1467] Processing CreateTablet for tablet 45de63e3c7cf4efbabe2f76d52c14f50 (DEFAULT_TABLE table=test-table [id=80417385beb24119a8fd0bd0bb9d2324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:36.094214 25080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45de63e3c7cf4efbabe2f76d52c14f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:36.097205 25498 raft_consensus.cc:2229] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:36.097620 25498 raft_consensus.cc:2258] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:36.099594 25498 ts_tablet_manager.cc:1905] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:36.103181 25501 tablet_bootstrap.cc:492] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: Bootstrap starting.
I20260928 15:50:36.105108 25501 tablet_bootstrap.cc:654] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:36.107115 25498 ts_tablet_manager.cc:1918] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:50:36.107280 25498 log.cc:1196] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/422f668ee87e49e5bcb7e6eb9c6cbf21
I20260928 15:50:36.107822 25498 ts_tablet_manager.cc:1939] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ed7292110a4e46f7980e0adbc7967bcf: Deleting consensus metadata
I20260928 15:50:36.107951 25501 tablet_bootstrap.cc:492] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: No bootstrap required, opened a new log
I20260928 15:50:36.108114 25501 ts_tablet_manager.cc:1397] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 15:50:36.107860 25362 tablet_service.cc:1467] Processing CreateTablet for tablet 45de63e3c7cf4efbabe2f76d52c14f50 (DEFAULT_TABLE table=test-table [id=80417385beb24119a8fd0bd0bb9d2324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:36.108639 25362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45de63e3c7cf4efbabe2f76d52c14f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:36.108952 25501 raft_consensus.cc:348] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:36.109231 25501 raft_consensus.cc:374] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:36.109336 25501 raft_consensus.cc:729] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa34856ffc7d44e4932d9acb6451a0bd, State: Initialized, Role: FOLLOWER
I20260928 15:50:36.109720 24928 catalog_manager.cc:4841] TS ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251): tablet 422f668ee87e49e5bcb7e6eb9c6cbf21 (table test-table [id=8d17ee38f82443ed9cc1fddce51deef3]) successfully deleted
I20260928 15:50:36.109695 25501 consensus_queue.cc:260] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:36.110312 25501 ts_tablet_manager.cc:1428] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:50:36.110410 25499 ts_tablet_manager.cc:1918] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:50:36.110625 25499 log.cc:1196] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/422f668ee87e49e5bcb7e6eb9c6cbf21
I20260928 15:50:36.111166 25499 ts_tablet_manager.cc:1939] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P ff1f5754815c45f1a3545c76e803afa1: Deleting consensus metadata
I20260928 15:50:36.112887 24929 catalog_manager.cc:4841] TS ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469): tablet 422f668ee87e49e5bcb7e6eb9c6cbf21 (table test-table [id=8d17ee38f82443ed9cc1fddce51deef3]) successfully deleted
I20260928 15:50:36.116745 25500 ts_tablet_manager.cc:1918] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:50:36.116931 25500 log.cc:1196] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/422f668ee87e49e5bcb7e6eb9c6cbf21
I20260928 15:50:36.117210 25503 tablet_bootstrap.cc:492] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: Bootstrap starting.
I20260928 15:50:36.117515 25500 ts_tablet_manager.cc:1939] T 422f668ee87e49e5bcb7e6eb9c6cbf21 P aa34856ffc7d44e4932d9acb6451a0bd: Deleting consensus metadata
I20260928 15:50:36.119356 24926 catalog_manager.cc:4841] TS aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194:38869): tablet 422f668ee87e49e5bcb7e6eb9c6cbf21 (table test-table [id=8d17ee38f82443ed9cc1fddce51deef3]) successfully deleted
I20260928 15:50:36.120144 25503 tablet_bootstrap.cc:654] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:36.122803 25503 tablet_bootstrap.cc:492] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: No bootstrap required, opened a new log
I20260928 15:50:36.122960 25503 ts_tablet_manager.cc:1397] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 15:50:36.124301 25503 raft_consensus.cc:348] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:36.124821 25503 raft_consensus.cc:374] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:36.124977 25503 raft_consensus.cc:729] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1f5754815c45f1a3545c76e803afa1, State: Initialized, Role: FOLLOWER
I20260928 15:50:36.125051 25505 tablet_bootstrap.cc:492] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: Bootstrap starting.
I20260928 15:50:36.125341 25503 consensus_queue.cc:260] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:36.126118 25503 ts_tablet_manager.cc:1428] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 15:50:36.126893 25505 tablet_bootstrap.cc:654] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:36.129402 25505 tablet_bootstrap.cc:492] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: No bootstrap required, opened a new log
I20260928 15:50:36.129616 25505 ts_tablet_manager.cc:1397] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 15:50:36.130517 25505 raft_consensus.cc:348] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:36.130762 25505 raft_consensus.cc:374] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:36.130856 25505 raft_consensus.cc:729] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7292110a4e46f7980e0adbc7967bcf, State: Initialized, Role: FOLLOWER
I20260928 15:50:36.131116 25505 consensus_queue.cc:260] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:36.131631 25505 ts_tablet_manager.cc:1428] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 15:50:36.168823 25486 raft_consensus.cc:482] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:36.169099 25486 raft_consensus.cc:504] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:36.171145 25486 leader_election.cc:288] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469), ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251)
I20260928 15:50:36.190621 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:36.172062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:41919 (local address 127.22.157.193:33469)
0928 15:50:36.172283 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:36.172290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:36.182881 (+ 10591us) server_negotiation.cc:407] Connection header received
0928 15:50:36.183264 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:36.183272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:36.183373 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:36.183449 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:36.184377 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.185557 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.188110 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.189542 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.189734 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.189810 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:36.190203 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 15:50:36.190333 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:50:36.190724 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:36.171537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:33469 (local address 127.22.157.194:41919)
0928 15:50:36.182724 (+ 11187us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:36.182754 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:36.183061 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:36.183681 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:36.183688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:36.183736 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:36.184163 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.184174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.185708 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.185715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.187933 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.187950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.189759 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.189766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.189948 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.190006 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 15:50:36.190153 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 15:50:36.190294 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10965,"mutex_wait_us":29,"thread_start_us":10755,"threads_started":1}
I20260928 15:50:36.190909 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:36.171605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:46251 (local address 127.22.157.194:41817)
0928 15:50:36.183991 (+ 12386us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:36.184015 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:36.184151 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:36.185109 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:36.185115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:36.185141 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:36.185451 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.185460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.187003 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.187008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.188701 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.188712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.190022 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.190035 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.190211 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.190244 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:50:36.190336 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:50:36.190471 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12231,"mutex_wait_us":1,"thread_start_us":11873,"threads_started":1}
I20260928 15:50:36.191624 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:36.171838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:41817 (local address 127.22.157.195:46251)
0928 15:50:36.184473 (+ 12635us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:36.184477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:36.184499 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:50:36.184586 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:36.184590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:36.184660 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:36.184921 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:36.185626 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.186825 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.188900 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.189814 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.189990 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.190058 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:36.191133 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0928 15:50:36.191282 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12503,"thread_start_us":11545,"threads_started":1}
I20260928 15:50:36.192025 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45de63e3c7cf4efbabe2f76d52c14f50" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1f5754815c45f1a3545c76e803afa1" is_pre_election: true
I20260928 15:50:36.192286 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45de63e3c7cf4efbabe2f76d52c14f50" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf" is_pre_election: true
I20260928 15:50:36.192518 25382 raft_consensus.cc:2454] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 0.
I20260928 15:50:36.192555 25100 raft_consensus.cc:2454] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 0.
I20260928 15:50:36.193132 25172 leader_election.cc:302] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ed7292110a4e46f7980e0adbc7967bcf; no voters:
I20260928 15:50:36.193609 25486 raft_consensus.cc:2781] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:36.193773 25486 raft_consensus.cc:482] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:36.193919 25486 raft_consensus.cc:3037] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:36.196216 25486 raft_consensus.cc:504] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:36.196954 25486 leader_election.cc:288] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 election: Requested vote from peers ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469), ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251)
I20260928 15:50:36.197525 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45de63e3c7cf4efbabe2f76d52c14f50" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1f5754815c45f1a3545c76e803afa1"
I20260928 15:50:36.197615 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45de63e3c7cf4efbabe2f76d52c14f50" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf"
I20260928 15:50:36.197794 25100 raft_consensus.cc:3037] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:36.197826 25382 raft_consensus.cc:3037] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:36.201082 25382 raft_consensus.cc:2454] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 1.
I20260928 15:50:36.201285 25100 raft_consensus.cc:2454] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 1.
I20260928 15:50:36.201656 25172 leader_election.cc:302] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ed7292110a4e46f7980e0adbc7967bcf; no voters:
I20260928 15:50:36.201997 25486 raft_consensus.cc:2781] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:36.202208 25486 raft_consensus.cc:686] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 LEADER]: Becoming Leader. State: Replica: aa34856ffc7d44e4932d9acb6451a0bd, State: Running, Role: LEADER
I20260928 15:50:36.202651 25486 consensus_queue.cc:237] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:36.207000 24941 catalog_manager.cc:5495] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd reported cstate change: term changed from 0 to 1, leader changed from <none> to aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194). New cstate: current_term: 1 leader_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } health_report { overall_health: UNKNOWN } } }
I20260928 15:50:36.259631 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:36.248341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.22.157.194:38869)
0928 15:50:36.253257 (+ 4916us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:36.253262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:36.253289 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:50:36.253397 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:36.253402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:36.253518 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:36.253717 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:36.254169 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.255404 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.256793 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.257466 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.257613 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.258100 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:36.258915 (+ 815us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:36.259082 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:36.259156 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 15:50:36.259242 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4760,"thread_start_us":4595,"threads_started":1}
I20260928 15:50:36.276203 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:36.270492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:55061 (local address 127.22.157.193:33469)
0928 15:50:36.270738 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:36.270745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:36.271177 (+ 432us) server_negotiation.cc:407] Connection header received
0928 15:50:36.271372 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:36.271380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:36.271539 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:36.271654 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:36.272204 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.273030 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.274391 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.274840 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.275001 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.275411 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:36.275510 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:36.275608 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:36.275744 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 15:50:36.275836 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:50:36.276247 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:36.270273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:33469 (local address 127.22.157.194:55061)
0928 15:50:36.271055 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:36.271070 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:36.271225 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:36.271688 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:36.271691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:36.271702 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:50:36.272001 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.272009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.273164 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.273170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.274236 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.274247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.275023 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.275029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.275172 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.275224 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:50:36.275608 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:50:36.275622 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:50:36.275721 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 15:50:36.275859 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":443,"threads_started":1}
I20260928 15:50:36.286936 25382 raft_consensus.cc:1261] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Refusing update from remote peer aa34856ffc7d44e4932d9acb6451a0bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:36.287554 25100 raft_consensus.cc:1261] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Refusing update from remote peer aa34856ffc7d44e4932d9acb6451a0bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:36.287648 25486 consensus_queue.cc:1035] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:36.288313 25486 consensus_queue.cc:1035] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:36.344074 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:36.334774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:38869 (local address 127.22.157.193:47053)
0928 15:50:36.335080 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:36.335100 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:36.335214 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:36.335785 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:36.335790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:36.335803 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:50:36.336090 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.336109 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.339847 (+ 3738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.339853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.341658 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:36.341676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.342702 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.342711 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:36.342926 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.343007 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:50:36.343444 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:50:36.343456 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 15:50:36.343544 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 15:50:36.343719 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260928 15:50:36.344853 25511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:36.334724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:47053 (local address 127.22.157.194:38869)
0928 15:50:36.334911 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:36.334917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:36.335355 (+ 438us) server_negotiation.cc:407] Connection header received
0928 15:50:36.335431 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:36.335436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:36.335524 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:36.335609 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:36.336366 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.337581 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.341823 (+ 4242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:36.342465 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:36.342604 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:36.343179 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:36.343307 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:36.343393 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:36.344382 (+ 989us) server_negotiation.cc:299] Negotiation successful
0928 15:50:36.344466 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 15:50:36.346028 25482 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 15:50:37.947345 24942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:37.947599 24942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:37.951668 24942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd: Sending DeleteTablet for 3 replicas of tablet 45de63e3c7cf4efbabe2f76d52c14f50
I20260928 15:50:37.952487 25221 tablet_service.cc:1514] Processing DeleteTablet for tablet 45de63e3c7cf4efbabe2f76d52c14f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:37 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:54918
I20260928 15:50:37.952544 25079 tablet_service.cc:1514] Processing DeleteTablet for tablet 45de63e3c7cf4efbabe2f76d52c14f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:37 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:55108
I20260928 15:50:37.952603 25362 tablet_service.cc:1514] Processing DeleteTablet for tablet 45de63e3c7cf4efbabe2f76d52c14f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:37 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:59658
I20260928 15:50:37.965219 25538 tablet_replica.cc:330] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: stopping tablet replica
I20260928 15:50:37.965632 25538 raft_consensus.cc:2229] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:50:37.966488 25538 raft_consensus.cc:2258] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:37.967231 25540 tablet_replica.cc:330] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: stopping tablet replica
I20260928 15:50:37.967782 25539 tablet_replica.cc:330] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: stopping tablet replica
I20260928 15:50:37.967826 25538 ts_tablet_manager.cc:1905] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:37.967993 25539 raft_consensus.cc:2229] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:37.968250 25539 raft_consensus.cc:2258] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:37.968600 25540 raft_consensus.cc:2229] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:37.968921 25540 raft_consensus.cc:2258] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:37.969695 25539 ts_tablet_manager.cc:1905] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:37.970547 25540 ts_tablet_manager.cc:1905] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:37.974529 25538 ts_tablet_manager.cc:1918] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:50:37.974750 25538 log.cc:1196] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/45de63e3c7cf4efbabe2f76d52c14f50
I20260928 15:50:37.975293 25538 ts_tablet_manager.cc:1939] T 45de63e3c7cf4efbabe2f76d52c14f50 P aa34856ffc7d44e4932d9acb6451a0bd: Deleting consensus metadata
I20260928 15:50:37.976600 24926 catalog_manager.cc:4841] TS aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194:38869): tablet 45de63e3c7cf4efbabe2f76d52c14f50 (table test-table [id=80417385beb24119a8fd0bd0bb9d2324]) successfully deleted
I20260928 15:50:37.977399 25540 ts_tablet_manager.cc:1918] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:50:37.977552 25540 log.cc:1196] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/45de63e3c7cf4efbabe2f76d52c14f50
I20260928 15:50:37.978047 25540 ts_tablet_manager.cc:1939] T 45de63e3c7cf4efbabe2f76d52c14f50 P ed7292110a4e46f7980e0adbc7967bcf: Deleting consensus metadata
I20260928 15:50:37.979087 25539 ts_tablet_manager.cc:1918] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:50:37.979257 25539 log.cc:1196] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/45de63e3c7cf4efbabe2f76d52c14f50
I20260928 15:50:37.979753 25539 ts_tablet_manager.cc:1939] T 45de63e3c7cf4efbabe2f76d52c14f50 P ff1f5754815c45f1a3545c76e803afa1: Deleting consensus metadata
I20260928 15:50:37.979686 24928 catalog_manager.cc:4841] TS ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251): tablet 45de63e3c7cf4efbabe2f76d52c14f50 (table test-table [id=80417385beb24119a8fd0bd0bb9d2324]) successfully deleted
I20260928 15:50:37.980803 24929 catalog_manager.cc:4841] TS ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469): tablet 45de63e3c7cf4efbabe2f76d52c14f50 (table test-table [id=80417385beb24119a8fd0bd0bb9d2324]) successfully deleted
Sep 28 15:50:37 dist-test-slave-1vw0 krb5kdc[24897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:50:38 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:38.025740 25548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:38.000821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60900 (local address 127.22.157.254:44399)
0928 15:50:38.001456 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:38.001462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:38.001503 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:50:38.001660 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:38.001666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:38.001750 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:38.001871 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:38.003265 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:38.004100 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:38.005495 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:38.005955 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:38.006116 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:38.019939 (+ 13823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:38.019945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:38.019949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:38.020000 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:38.022869 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:38.023681 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:38.023685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:38.023688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:38.023823 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:38.024192 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:38.024195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:38.024197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:38.024475 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:38.024676 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:38.025217 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 15:50:38.025422 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":302,"threads_started":1}
I20260928 15:50:38.030340 24942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:50:38.031148 24942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:50:38.038460 25079 tablet_service.cc:1467] Processing CreateTablet for tablet 3133a2094fc6412bb0fcc0edc3dde09b (DEFAULT_TABLE table=test-table [id=e662fec8e80c4ccc9289d101e763be24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:38.038743 25221 tablet_service.cc:1467] Processing CreateTablet for tablet 3133a2094fc6412bb0fcc0edc3dde09b (DEFAULT_TABLE table=test-table [id=e662fec8e80c4ccc9289d101e763be24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:38.038931 25362 tablet_service.cc:1467] Processing CreateTablet for tablet 3133a2094fc6412bb0fcc0edc3dde09b (DEFAULT_TABLE table=test-table [id=e662fec8e80c4ccc9289d101e763be24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:38.039054 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3133a2094fc6412bb0fcc0edc3dde09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:38.039234 25221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3133a2094fc6412bb0fcc0edc3dde09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:38.039322 25362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3133a2094fc6412bb0fcc0edc3dde09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:38.046613 25549 tablet_bootstrap.cc:492] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: Bootstrap starting.
I20260928 15:50:38.048295 25549 tablet_bootstrap.cc:654] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:38.050972 25550 tablet_bootstrap.cc:492] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: Bootstrap starting.
I20260928 15:50:38.051743 25549 tablet_bootstrap.cc:492] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: No bootstrap required, opened a new log
I20260928 15:50:38.051944 25549 ts_tablet_manager.cc:1397] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 15:50:38.052812 25549 raft_consensus.cc:348] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.053329 25549 raft_consensus.cc:374] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:38.053344 25551 tablet_bootstrap.cc:492] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: Bootstrap starting.
I20260928 15:50:38.053938 25549 raft_consensus.cc:729] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1f5754815c45f1a3545c76e803afa1, State: Initialized, Role: FOLLOWER
I20260928 15:50:38.054327 25549 consensus_queue.cc:260] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.055049 25549 ts_tablet_manager.cc:1428] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 15:50:38.055464 25551 tablet_bootstrap.cc:654] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:38.057382 25550 tablet_bootstrap.cc:654] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:38.058012 25551 tablet_bootstrap.cc:492] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: No bootstrap required, opened a new log
I20260928 15:50:38.058193 25551 ts_tablet_manager.cc:1397] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 15:50:38.059170 25551 raft_consensus.cc:348] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.059592 25551 raft_consensus.cc:374] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:38.059743 25551 raft_consensus.cc:729] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa34856ffc7d44e4932d9acb6451a0bd, State: Initialized, Role: FOLLOWER
I20260928 15:50:38.060024 25551 consensus_queue.cc:260] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.060549 25551 ts_tablet_manager.cc:1428] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 15:50:38.061545 25550 tablet_bootstrap.cc:492] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: No bootstrap required, opened a new log
I20260928 15:50:38.061710 25550 ts_tablet_manager.cc:1397] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260928 15:50:38.062595 25550 raft_consensus.cc:348] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.062868 25550 raft_consensus.cc:374] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:38.062975 25550 raft_consensus.cc:729] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7292110a4e46f7980e0adbc7967bcf, State: Initialized, Role: FOLLOWER
I20260928 15:50:38.063236 25550 consensus_queue.cc:260] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.064037 25550 ts_tablet_manager.cc:1428] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 15:50:38.131970 25486 raft_consensus.cc:482] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:38.132242 25486 raft_consensus.cc:504] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.133050 25486 leader_election.cc:288] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469), ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251)
I20260928 15:50:38.133781 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3133a2094fc6412bb0fcc0edc3dde09b" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1f5754815c45f1a3545c76e803afa1" is_pre_election: true
I20260928 15:50:38.133781 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3133a2094fc6412bb0fcc0edc3dde09b" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf" is_pre_election: true
I20260928 15:50:38.134068 25382 raft_consensus.cc:2454] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 0.
I20260928 15:50:38.134109 25100 raft_consensus.cc:2454] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 0.
I20260928 15:50:38.134651 25172 leader_election.cc:302] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ed7292110a4e46f7980e0adbc7967bcf; no voters:
I20260928 15:50:38.134979 25486 raft_consensus.cc:2781] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:38.135120 25486 raft_consensus.cc:482] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:38.135258 25486 raft_consensus.cc:3037] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:38.137756 25486 raft_consensus.cc:504] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.138397 25486 leader_election.cc:288] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 election: Requested vote from peers ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469), ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251)
I20260928 15:50:38.139074 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3133a2094fc6412bb0fcc0edc3dde09b" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1f5754815c45f1a3545c76e803afa1"
I20260928 15:50:38.139037 25382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3133a2094fc6412bb0fcc0edc3dde09b" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf"
I20260928 15:50:38.139297 25100 raft_consensus.cc:3037] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:38.139297 25382 raft_consensus.cc:3037] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:38.142406 25382 raft_consensus.cc:2454] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 1.
I20260928 15:50:38.142405 25100 raft_consensus.cc:2454] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 1.
I20260928 15:50:38.142935 25173 leader_election.cc:302] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ff1f5754815c45f1a3545c76e803afa1; no voters:
I20260928 15:50:38.143288 25486 raft_consensus.cc:2781] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:38.143467 25486 raft_consensus.cc:686] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 1 LEADER]: Becoming Leader. State: Replica: aa34856ffc7d44e4932d9acb6451a0bd, State: Running, Role: LEADER
I20260928 15:50:38.143793 25486 consensus_queue.cc:237] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.146646 24941 catalog_manager.cc:5495] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd reported cstate change: term changed from 0 to 1, leader changed from <none> to aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194). New cstate: current_term: 1 leader_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } health_report { overall_health: UNKNOWN } } }
I20260928 15:50:38.188570 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:38.183046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.22.157.194:38869)
0928 15:50:38.183674 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:38.183680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:38.183701 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:50:38.183768 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:38.183771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:38.183834 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:38.183943 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:38.184341 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:38.185164 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:38.186622 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:38.187086 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:38.187228 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:38.187749 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:38.187875 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:38.187977 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:38.188161 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 15:50:38.188250 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":331,"threads_started":1}
I20260928 15:50:38.198697 25382 raft_consensus.cc:1261] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Refusing update from remote peer aa34856ffc7d44e4932d9acb6451a0bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:38.198807 25100 raft_consensus.cc:1261] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Refusing update from remote peer aa34856ffc7d44e4932d9acb6451a0bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:38.199404 25486 consensus_queue.cc:1035] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:38.199728 25530 consensus_queue.cc:1035] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:38.206893 25521 mvcc.cc:205] Tried to move back new op lower bound from 7334341174056898560 to 7334341173839405056. Current Snapshot: MvccSnapshot[applied={T|T < 7334341174056898560}]
I20260928 15:50:38.254788 24942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:38.255131 24942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:38.259073 24942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd: Sending DeleteTablet for 3 replicas of tablet 3133a2094fc6412bb0fcc0edc3dde09b
I20260928 15:50:38.260102 25221 tablet_service.cc:1514] Processing DeleteTablet for tablet 3133a2094fc6412bb0fcc0edc3dde09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:38 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:54918
I20260928 15:50:38.260123 25079 tablet_service.cc:1514] Processing DeleteTablet for tablet 3133a2094fc6412bb0fcc0edc3dde09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:38 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:55108
I20260928 15:50:38.260175 25362 tablet_service.cc:1514] Processing DeleteTablet for tablet 3133a2094fc6412bb0fcc0edc3dde09b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:38 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:59658
I20260928 15:50:38.260437 25538 tablet_replica.cc:330] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: stopping tablet replica
I20260928 15:50:38.260439 25540 tablet_replica.cc:330] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: stopping tablet replica
I20260928 15:50:38.260453 25539 tablet_replica.cc:330] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: stopping tablet replica
I20260928 15:50:38.260645 25538 raft_consensus.cc:2229] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:50:38.260651 25539 raft_consensus.cc:2229] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:38.260649 25540 raft_consensus.cc:2229] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:38.260878 25539 raft_consensus.cc:2258] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:38.260878 25540 raft_consensus.cc:2258] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:38.260993 25538 raft_consensus.cc:2258] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:38.262212 25539 ts_tablet_manager.cc:1905] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:38.262420 25540 ts_tablet_manager.cc:1905] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:38.262418 24942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 15:50:38.262706 25538 ts_tablet_manager.cc:1905] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 15:50:38.263121 24942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:50:38.269271 25540 ts_tablet_manager.cc:1918] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:50:38.269460 25538 ts_tablet_manager.cc:1918] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:50:38.269536 25540 log.cc:1196] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/3133a2094fc6412bb0fcc0edc3dde09b
I20260928 15:50:38.269690 25538 log.cc:1196] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/3133a2094fc6412bb0fcc0edc3dde09b
I20260928 15:50:38.269805 25539 ts_tablet_manager.cc:1918] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:50:38.270040 25539 log.cc:1196] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/3133a2094fc6412bb0fcc0edc3dde09b
I20260928 15:50:38.270151 25538 ts_tablet_manager.cc:1939] T 3133a2094fc6412bb0fcc0edc3dde09b P aa34856ffc7d44e4932d9acb6451a0bd: Deleting consensus metadata
I20260928 15:50:38.270306 25540 ts_tablet_manager.cc:1939] T 3133a2094fc6412bb0fcc0edc3dde09b P ed7292110a4e46f7980e0adbc7967bcf: Deleting consensus metadata
I20260928 15:50:38.270533 25539 ts_tablet_manager.cc:1939] T 3133a2094fc6412bb0fcc0edc3dde09b P ff1f5754815c45f1a3545c76e803afa1: Deleting consensus metadata
I20260928 15:50:38.271476 24928 catalog_manager.cc:4841] TS ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251): tablet 3133a2094fc6412bb0fcc0edc3dde09b (table test-table [id=e662fec8e80c4ccc9289d101e763be24]) successfully deleted
I20260928 15:50:38.271723 24926 catalog_manager.cc:4841] TS aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194:38869): tablet 3133a2094fc6412bb0fcc0edc3dde09b (table test-table [id=e662fec8e80c4ccc9289d101e763be24]) successfully deleted
I20260928 15:50:38.271698 25362 tablet_service.cc:1467] Processing CreateTablet for tablet 91e4c8f8b7604232ad1673d853017711 (DEFAULT_TABLE table=test-table [id=5ac2f62691c94491af92d70ec55d1387]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:38.272193 25362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e4c8f8b7604232ad1673d853017711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:38.272176 25221 tablet_service.cc:1467] Processing CreateTablet for tablet 91e4c8f8b7604232ad1673d853017711 (DEFAULT_TABLE table=test-table [id=5ac2f62691c94491af92d70ec55d1387]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:38.272588 25221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e4c8f8b7604232ad1673d853017711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:38.272652 25079 tablet_service.cc:1467] Processing CreateTablet for tablet 91e4c8f8b7604232ad1673d853017711 (DEFAULT_TABLE table=test-table [id=5ac2f62691c94491af92d70ec55d1387]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:38.273064 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e4c8f8b7604232ad1673d853017711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:38.279023 25551 tablet_bootstrap.cc:492] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: Bootstrap starting.
I20260928 15:50:38.279245 25550 tablet_bootstrap.cc:492] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: Bootstrap starting.
I20260928 15:50:38.280783 25551 tablet_bootstrap.cc:654] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:38.280862 25550 tablet_bootstrap.cc:654] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:38.282042 25549 tablet_bootstrap.cc:492] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: Bootstrap starting.
I20260928 15:50:38.283077 25551 tablet_bootstrap.cc:492] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: No bootstrap required, opened a new log
I20260928 15:50:38.283237 25551 ts_tablet_manager.cc:1397] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 15:50:38.283569 25549 tablet_bootstrap.cc:654] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:38.284008 25551 raft_consensus.cc:348] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.284253 25551 raft_consensus.cc:374] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:38.284324 25551 raft_consensus.cc:729] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa34856ffc7d44e4932d9acb6451a0bd, State: Initialized, Role: FOLLOWER
I20260928 15:50:38.284649 25551 consensus_queue.cc:260] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.285295 25551 ts_tablet_manager.cc:1428] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:50:38.286064 24929 catalog_manager.cc:4841] TS ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469): tablet 3133a2094fc6412bb0fcc0edc3dde09b (table test-table [id=e662fec8e80c4ccc9289d101e763be24]) successfully deleted
I20260928 15:50:38.287406 25550 tablet_bootstrap.cc:492] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: No bootstrap required, opened a new log
I20260928 15:50:38.287424 25549 tablet_bootstrap.cc:492] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: No bootstrap required, opened a new log
I20260928 15:50:38.287592 25549 ts_tablet_manager.cc:1397] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260928 15:50:38.287674 25550 ts_tablet_manager.cc:1397] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260928 15:50:38.288443 25549 raft_consensus.cc:348] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.288458 25550 raft_consensus.cc:348] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.288690 25549 raft_consensus.cc:374] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:38.288743 25550 raft_consensus.cc:374] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:38.288815 25549 raft_consensus.cc:729] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1f5754815c45f1a3545c76e803afa1, State: Initialized, Role: FOLLOWER
I20260928 15:50:38.288822 25550 raft_consensus.cc:729] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7292110a4e46f7980e0adbc7967bcf, State: Initialized, Role: FOLLOWER
I20260928 15:50:38.289103 25549 consensus_queue.cc:260] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.289103 25550 consensus_queue.cc:260] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.289801 25550 ts_tablet_manager.cc:1428] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 15:50:38.290089 25549 ts_tablet_manager.cc:1428] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:50:38.513566 25562 raft_consensus.cc:482] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:38.513834 25562 raft_consensus.cc:504] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.514614 25562 leader_election.cc:288] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469), ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251)
I20260928 15:50:38.515232 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e4c8f8b7604232ad1673d853017711" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf" is_pre_election: true
I20260928 15:50:38.515254 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e4c8f8b7604232ad1673d853017711" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1f5754815c45f1a3545c76e803afa1" is_pre_election: true
I20260928 15:50:38.515537 25381 raft_consensus.cc:2454] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 0.
I20260928 15:50:38.515582 25100 raft_consensus.cc:2454] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 0.
I20260928 15:50:38.516005 25172 leader_election.cc:302] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ed7292110a4e46f7980e0adbc7967bcf; no voters:
I20260928 15:50:38.516299 25562 raft_consensus.cc:2781] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:38.516405 25562 raft_consensus.cc:482] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:38.516541 25562 raft_consensus.cc:3037] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:38.518734 25562 raft_consensus.cc:504] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.519371 25562 leader_election.cc:288] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 election: Requested vote from peers ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469), ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251)
I20260928 15:50:38.519914 25100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e4c8f8b7604232ad1673d853017711" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff1f5754815c45f1a3545c76e803afa1"
I20260928 15:50:38.519966 25381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e4c8f8b7604232ad1673d853017711" candidate_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7292110a4e46f7980e0adbc7967bcf"
I20260928 15:50:38.520147 25100 raft_consensus.cc:3037] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:38.520246 25381 raft_consensus.cc:3037] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:38.522361 25100 raft_consensus.cc:2454] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 1.
I20260928 15:50:38.522420 25381 raft_consensus.cc:2454] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa34856ffc7d44e4932d9acb6451a0bd in term 1.
I20260928 15:50:38.522846 25173 leader_election.cc:302] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa34856ffc7d44e4932d9acb6451a0bd, ff1f5754815c45f1a3545c76e803afa1; no voters:
I20260928 15:50:38.523217 25562 raft_consensus.cc:2781] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:38.523402 25562 raft_consensus.cc:686] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 LEADER]: Becoming Leader. State: Replica: aa34856ffc7d44e4932d9acb6451a0bd, State: Running, Role: LEADER
I20260928 15:50:38.523720 25562 consensus_queue.cc:237] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } }
I20260928 15:50:38.526778 24940 catalog_manager.cc:5495] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd reported cstate change: term changed from 0 to 1, leader changed from <none> to aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194). New cstate: current_term: 1 leader_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa34856ffc7d44e4932d9acb6451a0bd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 } health_report { overall_health: UNKNOWN } } }
I20260928 15:50:38.552059 25382 raft_consensus.cc:1261] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Refusing update from remote peer aa34856ffc7d44e4932d9acb6451a0bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:38.552644 25100 raft_consensus.cc:1261] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Refusing update from remote peer aa34856ffc7d44e4932d9acb6451a0bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:38.552616 25530 consensus_queue.cc:1035] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7292110a4e46f7980e0adbc7967bcf" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 46251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:38.553217 25530 consensus_queue.cc:1035] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff1f5754815c45f1a3545c76e803afa1" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:50:38.574884 25522 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 15:50:40.163707 24942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:40.163993 24942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:40.168188 24942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 088891443ba343ae971692d302adb6bd: Sending DeleteTablet for 3 replicas of tablet 91e4c8f8b7604232ad1673d853017711
I20260928 15:50:40.169137 25079 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e4c8f8b7604232ad1673d853017711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:40 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:55108
I20260928 15:50:40.169123 25221 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e4c8f8b7604232ad1673d853017711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:40 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:54918
I20260928 15:50:40.169212 25362 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e4c8f8b7604232ad1673d853017711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:40 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:59658
I20260928 15:50:40.169934 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25014
I20260928 15:50:40.181660 25591 tablet_replica.cc:330] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: stopping tablet replica
I20260928 15:50:40.181869 25589 tablet_replica.cc:330] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: stopping tablet replica
I20260928 15:50:40.182133 25591 raft_consensus.cc:2229] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:40.182433 25589 raft_consensus.cc:2229] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:40.182572 25591 raft_consensus.cc:2258] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:40.182848 25589 raft_consensus.cc:2258] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:40.184211 25589 ts_tablet_manager.cc:1905] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:40.184222 25591 ts_tablet_manager.cc:1905] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:40.193212 25590 tablet_replica.cc:330] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: stopping tablet replica
I20260928 15:50:40.193496 25590 raft_consensus.cc:2229] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:50:40.194061 25590 raft_consensus.cc:2258] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:40.194296 25591 ts_tablet_manager.cc:1918] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:50:40.194422 25589 ts_tablet_manager.cc:1918] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:50:40.194507 25591 log.cc:1196] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/91e4c8f8b7604232ad1673d853017711
I20260928 15:50:40.194602 25589 log.cc:1196] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/91e4c8f8b7604232ad1673d853017711
I20260928 15:50:40.195078 25589 ts_tablet_manager.cc:1939] T 91e4c8f8b7604232ad1673d853017711 P ff1f5754815c45f1a3545c76e803afa1: Deleting consensus metadata
I20260928 15:50:40.195253 25591 ts_tablet_manager.cc:1939] T 91e4c8f8b7604232ad1673d853017711 P ed7292110a4e46f7980e0adbc7967bcf: Deleting consensus metadata
I20260928 15:50:40.195314 25590 ts_tablet_manager.cc:1905] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:40.196272 24929 catalog_manager.cc:4841] TS ff1f5754815c45f1a3545c76e803afa1 (127.22.157.193:33469): tablet 91e4c8f8b7604232ad1673d853017711 (table test-table [id=5ac2f62691c94491af92d70ec55d1387]) successfully deleted
I20260928 15:50:40.196425 24928 catalog_manager.cc:4841] TS ed7292110a4e46f7980e0adbc7967bcf (127.22.157.195:46251): tablet 91e4c8f8b7604232ad1673d853017711 (table test-table [id=5ac2f62691c94491af92d70ec55d1387]) successfully deleted
I20260928 15:50:40.202474 25590 ts_tablet_manager.cc:1918] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 15:50:40.202651 25590 log.cc:1196] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/91e4c8f8b7604232ad1673d853017711
I20260928 15:50:40.203135 25590 ts_tablet_manager.cc:1939] T 91e4c8f8b7604232ad1673d853017711 P aa34856ffc7d44e4932d9acb6451a0bd: Deleting consensus metadata
Sep 28 15:50:40 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:50:40.204274 24926 catalog_manager.cc:4841] TS aa34856ffc7d44e4932d9acb6451a0bd (127.22.157.194:38869): tablet 91e4c8f8b7604232ad1673d853017711 (table test-table [id=5ac2f62691c94491af92d70ec55d1387]) successfully deleted
I20260928 15:50:40.211282 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:40.172440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.22.157.193:33469)
0928 15:50:40.193213 (+ 20773us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:40.193218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:40.193248 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:50:40.193495 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:40.193501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:40.193617 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:40.193705 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:40.195772 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:40.196920 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:40.198760 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:40.199194 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:40.199291 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:40.204606 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:40.204625 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:40.204641 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:40.204681 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:40.207997 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:40.208828 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:40.208837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:40.208839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:40.208990 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:40.209432 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:40.209438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:40.209443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:40.209871 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:40.210231 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:40.210595 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 15:50:40.210849 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20598,"thread_start_us":20457,"threads_started":1}
W20260928 15:50:40.212036 25145 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55124
W20260928 15:50:40.212790 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25014: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:40.213042 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 25014
I20260928 15:50:40.241456 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25154
Sep 28 15:50:40 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:50:40.258144 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:40.242337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.22.157.194:38869)
0928 15:50:40.242838 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:40.242843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:40.242859 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:50:40.242926 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:40.242929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:40.243027 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:40.243179 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:40.244458 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:40.245304 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:40.246615 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:40.247048 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:40.247149 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:40.251690 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:40.251709 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:40.251725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:40.251771 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:40.254795 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:40.255558 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:40.255579 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:40.255582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:40.255729 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:40.256122 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:40.256128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:40.256132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:40.256533 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:40.256931 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:40.257370 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 15:50:40.257631 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":264,"threads_started":1}
W20260928 15:50:40.258824 25287 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54944
W20260928 15:50:40.259501 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25154: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:40.259635 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 25154
I20260928 15:50:40.282029 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25296
Sep 28 15:50:40 dist-test-slave-1vw0 krb5kdc[24897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610637, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:50:40.299849 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:40.282839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.22.157.195:46251)
0928 15:50:40.283471 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:40.283479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:40.283503 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:50:40.283574 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:40.283580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:40.283669 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:40.283794 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:40.285475 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:40.286754 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:40.288031 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:40.288879 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:40.289019 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:40.293904 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:40.293922 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:40.293939 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:40.293985 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:40.297206 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:40.297935 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:40.297943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:40.297945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:40.298079 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:40.298403 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:40.298406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:40.298408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:40.298800 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:40.299057 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:40.299356 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 15:50:40.299586 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":290,"threads_started":1}
W20260928 15:50:40.300627 25427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59662
W20260928 15:50:40.301491 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25296: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:40.301685 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 25296
I20260928 15:50:40.320276 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 24910
I20260928 15:50:40.321537 24982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:40.441002 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 24910
2026-09-28T15:50:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8597 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:40 dist-test-slave-1vw0 krb5kdc[25599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:40 dist-test-slave-1vw0 krb5kdc[25599](info): set up 2 sockets
Sep 28 15:50:40 dist-test-slave-1vw0 krb5kdc[25599](info): commencing operation
krb5kdc: starting...
W20260928 15:50:41.553527 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.007s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:50:41 dist-test-slave-1vw0 krb5kdc[25599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:50:41.764629 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45899
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:45899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:42.106065 25612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:42.106577 25612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:42.106765 25612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:42.115020 25612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:42.115168 25612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:42.115284 25612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:42.115396 25612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:42.115479 25612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:42.125838 25612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:45899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:42.127303 25612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:42.128772 25612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:42.141651 25617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:42.141619 25620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:42.141794 25612 server_base.cc:1034] running on GCE node
W20260928 15:50:42.141677 25618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:42.143054 25612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:42.144619 25612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:42.145946 25612 hybrid_clock.cc:648] HybridClock initialized: now 1790610642145831 us; error 39 us; skew 500 ppm
I20260928 15:50:42.146322 25612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:42 dist-test-slave-1vw0 krb5kdc[25599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610642, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:42.149637 25612 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:50:42.151378 25612 webserver.cc:460] Webserver started at http://127.22.157.254:40529/ using document root <none> and password file <none>
I20260928 15:50:42.152158 25612 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:42.152308 25612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:42.152648 25612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:42.155921 25612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "531d04b2c148462486ad94e0a9783178"
format_stamp: "Formatted at 2026-09-28 15:50:42 on dist-test-slave-1vw0"
server_key: "c4bda35a298d05a205cba840a0d663d5"
server_key_iv: "5b26ae88ae960beb15e3a8ab855af783"
server_key_version: "encryptionkey@0"
I20260928 15:50:42.156806 25612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "531d04b2c148462486ad94e0a9783178"
format_stamp: "Formatted at 2026-09-28 15:50:42 on dist-test-slave-1vw0"
server_key: "c4bda35a298d05a205cba840a0d663d5"
server_key_iv: "5b26ae88ae960beb15e3a8ab855af783"
server_key_version: "encryptionkey@0"
I20260928 15:50:42.162374 25612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:50:42.166136 25627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:42.167902 25612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 15:50:42.168107 25612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "531d04b2c148462486ad94e0a9783178"
format_stamp: "Formatted at 2026-09-28 15:50:42 on dist-test-slave-1vw0"
server_key: "c4bda35a298d05a205cba840a0d663d5"
server_key_iv: "5b26ae88ae960beb15e3a8ab855af783"
server_key_version: "encryptionkey@0"
I20260928 15:50:42.168318 25612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:42.188035 25612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:42.192073 25612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:42.192377 25612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:42.211833 25612 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:45899
I20260928 15:50:42.211850 25680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:45899 every 8 connection(s)
I20260928 15:50:42.213299 25612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:42.216967 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 25612
I20260928 15:50:42.217211 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:50:42.217495 25681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:42.218034 23159 external_mini_cluster.cc:1428] Setting key ee97897003a72f882fe1826a8afc49ff
I20260928 15:50:42.232846 25681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178: Bootstrap starting.
Sep 28 15:50:42 dist-test-slave-1vw0 krb5kdc[25599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:42.236835 25681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:42.238329 25681 log.cc:824] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:42.241514 25681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178: No bootstrap required, opened a new log
I20260928 15:50:42.243314 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:42.220224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.22.157.254:45899)
0928 15:50:42.221087 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:42.221101 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:42.221147 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:50:42.222806 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:42.222830 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:42.223261 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:42.223626 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:42.225598 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:42.226961 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:42.228835 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:42.229654 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:42.229809 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:42.235018 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:42.235026 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:42.235043 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:42.235097 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:42.239036 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:42.239807 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:42.239816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:42.239830 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:42.239998 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:42.240383 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:42.240389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:42.240392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:42.241012 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:42.241307 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:42.241814 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 15:50:42.242275 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":432,"threads_started":1}
I20260928 15:50:42.250597 25681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d04b2c148462486ad94e0a9783178" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45899 } }
I20260928 15:50:42.251025 25681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:42.251137 25681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531d04b2c148462486ad94e0a9783178, State: Initialized, Role: FOLLOWER
I20260928 15:50:42.251691 25681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d04b2c148462486ad94e0a9783178" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45899 } }
I20260928 15:50:42.251945 25681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:42.252087 25681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:42.252259 25681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:42.254858 25681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d04b2c148462486ad94e0a9783178" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45899 } }
I20260928 15:50:42.255395 25681 leader_election.cc:302] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531d04b2c148462486ad94e0a9783178; no voters:
I20260928 15:50:42.256026 25681 leader_election.cc:288] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:42.256186 25686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:42.256980 25686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [term 1 LEADER]: Becoming Leader. State: Replica: 531d04b2c148462486ad94e0a9783178, State: Running, Role: LEADER
I20260928 15:50:42.257548 25686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d04b2c148462486ad94e0a9783178" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45899 } }
I20260928 15:50:42.258241 25681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:42.260174 25688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531d04b2c148462486ad94e0a9783178. Latest consensus state: current_term: 1 leader_uuid: "531d04b2c148462486ad94e0a9783178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d04b2c148462486ad94e0a9783178" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45899 } } }
I20260928 15:50:42.260593 25688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:42.261159 25687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531d04b2c148462486ad94e0a9783178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d04b2c148462486ad94e0a9783178" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45899 } } }
I20260928 15:50:42.261642 25687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:42.266059 25691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:42.271826 25691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:42.281178 25691 catalog_manager.cc:1347] Generated new cluster ID: 4e7a3de212ec49218878b9c22101f68f
I20260928 15:50:42.281319 25691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:42.297812 25691 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:42.299161 25691 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:42.313629 25691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 531d04b2c148462486ad94e0a9783178: Generated new TSK 0
I20260928 15:50:42.314603 25691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:50:42.372908 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45899
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:42.727388 25708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:42.727756 25708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:42.727934 25708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:42.735544 25708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:42.735730 25708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:42.735975 25708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:50:42.746394 25708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:42.747661 25708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:42.749049 25708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:42.762997 25716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:42.763024 25713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:42.763012 25714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:42.763839 25708 server_base.cc:1034] running on GCE node
I20260928 15:50:42.764704 25708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:42.765880 25708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:42.767162 25708 hybrid_clock.cc:648] HybridClock initialized: now 1790610642767113 us; error 35 us; skew 500 ppm
I20260928 15:50:42.767639 25708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:42 dist-test-slave-1vw0 krb5kdc[25599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610642, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:42.771965 25708 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:50:42.773746 25708 webserver.cc:460] Webserver started at http://127.22.157.193:33157/ using document root <none> and password file <none>
I20260928 15:50:42.774456 25708 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:42.774560 25708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:42.774824 25708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:42.777947 25708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "0b849fc0471b4a8d9c8fa903e52c1c2a"
format_stamp: "Formatted at 2026-09-28 15:50:42 on dist-test-slave-1vw0"
server_key: "c32fbf71711d07ac9c79cfceb5736a4b"
server_key_iv: "0ee870512232e979a1d2d4820d32c588"
server_key_version: "encryptionkey@0"
I20260928 15:50:42.778729 25708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "0b849fc0471b4a8d9c8fa903e52c1c2a"
format_stamp: "Formatted at 2026-09-28 15:50:42 on dist-test-slave-1vw0"
server_key: "c32fbf71711d07ac9c79cfceb5736a4b"
server_key_iv: "0ee870512232e979a1d2d4820d32c588"
server_key_version: "encryptionkey@0"
I20260928 15:50:42.784312 25708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:50:42.787950 25723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:42.789402 25708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:42.789582 25708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "0b849fc0471b4a8d9c8fa903e52c1c2a"
format_stamp: "Formatted at 2026-09-28 15:50:42 on dist-test-slave-1vw0"
server_key: "c32fbf71711d07ac9c79cfceb5736a4b"
server_key_iv: "0ee870512232e979a1d2d4820d32c588"
server_key_version: "encryptionkey@0"
I20260928 15:50:42.789791 25708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:42.805820 25708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:42.810146 25708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:42.810459 25708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:42.811480 25708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:42.813131 25708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:42.813232 25708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:42.813385 25708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:42.813464 25708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:42.846849 25708 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:43367
I20260928 15:50:42.846906 25836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:43367 every 8 connection(s)
I20260928 15:50:42.848254 25708 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:50:42.856360 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 25708
I20260928 15:50:42.856626 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:50:42.857318 23159 external_mini_cluster.cc:1428] Setting key e905955b5b372d86b653e5e49f594061
Sep 28 15:50:42 dist-test-slave-1vw0 krb5kdc[25599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610642, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:42.870169 25839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:42.851148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:53293 (local address 127.22.157.254:45899)
0928 15:50:42.851673 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:42.851678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:42.852674 (+ 996us) server_negotiation.cc:407] Connection header received
0928 15:50:42.854599 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:42.854605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:42.854703 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:42.854825 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:42.856744 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:42.857567 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:42.859346 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:42.859904 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:42.860050 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:42.863827 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:42.863832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:42.863835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:42.863867 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:42.866942 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:42.867774 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:42.867781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:42.867790 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:42.867909 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:42.868341 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:42.868348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:42.868351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:42.868647 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:42.868820 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:42.869529 (+ 709us) server_negotiation.cc:299] Negotiation successful
0928 15:50:42.869794 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":265,"threads_started":1}
I20260928 15:50:42.871214 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:42.851762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45899 (local address 127.22.157.193:53293)
0928 15:50:42.852492 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:42.852536 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:42.854299 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:42.855087 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:42.855097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:42.855649 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:42.856503 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:42.856533 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:42.857743 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:42.857749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:42.859193 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:42.859205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:42.860162 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:42.860170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:42.860407 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:42.861137 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:42.861159 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:42.863604 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:42.867086 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:42.867095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:42.867113 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:42.867648 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:42.868023 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:42.868030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:42.868033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:42.868226 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:42.868804 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:42.868811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:42.869162 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 15:50:42.869436 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 15:50:42.870023 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":280,"threads_started":1}
I20260928 15:50:42.872963 25837 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45899
I20260928 15:50:42.873286 25837 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:42.874033 25837 heartbeater.cc:511] Master 127.22.157.254:45899 requested a full tablet report, sending...
I20260928 15:50:42.876121 25646 ts_manager.cc:194] Registered new tserver with Master: 0b849fc0471b4a8d9c8fa903e52c1c2a (127.22.157.193:43367)
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
I20260928 15:50:42.877681 25646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:53293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:50:42.904930 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45899
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:43.229585 25843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:43.229885 25843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:43.230054 25843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:43.237622 25843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:43.237744 25843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:43.237931 25843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:50:43.248114 25843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:43.249564 25843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:43.250948 25843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:43.264748 25851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:43.264675 25849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:43.264792 25848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:43.270398 25843 server_base.cc:1034] running on GCE node
I20260928 15:50:43.271157 25843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:43.272238 25843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:43.273507 25843 hybrid_clock.cc:648] HybridClock initialized: now 1790610643273436 us; error 61 us; skew 500 ppm
I20260928 15:50:43.273883 25843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:43 dist-test-slave-1vw0 krb5kdc[25599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610643, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:43.282805 25843 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:50:43.284626 25843 webserver.cc:460] Webserver started at http://127.22.157.194:44199/ using document root <none> and password file <none>
I20260928 15:50:43.285460 25843 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:43.285616 25843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:43.285955 25843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:43.289203 25843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "1a8e66fe7fbb4ba786ea71ae1c3a28a7"
format_stamp: "Formatted at 2026-09-28 15:50:43 on dist-test-slave-1vw0"
server_key: "fab416fb4a30153302672085b352a58b"
server_key_iv: "a0523e63a43068065dfa01824d4e1e60"
server_key_version: "encryptionkey@0"
I20260928 15:50:43.290047 25843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "1a8e66fe7fbb4ba786ea71ae1c3a28a7"
format_stamp: "Formatted at 2026-09-28 15:50:43 on dist-test-slave-1vw0"
server_key: "fab416fb4a30153302672085b352a58b"
server_key_iv: "a0523e63a43068065dfa01824d4e1e60"
server_key_version: "encryptionkey@0"
I20260928 15:50:43.295746 25843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:50:43.299602 25858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:43.301437 25843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:50:43.301595 25843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "1a8e66fe7fbb4ba786ea71ae1c3a28a7"
format_stamp: "Formatted at 2026-09-28 15:50:43 on dist-test-slave-1vw0"
server_key: "fab416fb4a30153302672085b352a58b"
server_key_iv: "a0523e63a43068065dfa01824d4e1e60"
server_key_version: "encryptionkey@0"
I20260928 15:50:43.301791 25843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:43.330794 25843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:43.335700 25843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:43.335981 25843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:43.337025 25843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:43.338676 25843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:43.338761 25843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:43.338856 25843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:43.338907 25843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:43.374785 25843 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:42573
I20260928 15:50:43.374819 25970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:42573 every 8 connection(s)
I20260928 15:50:43.376677 25843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:50:43.385620 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 25843
I20260928 15:50:43.385831 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:50:43.386359 23159 external_mini_cluster.cc:1428] Setting key d09e3cd1601a3f19284d0aaf99788fa1
Sep 28 15:50:43 dist-test-slave-1vw0 krb5kdc[25599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610643, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:43.399104 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:43.379262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:49581 (local address 127.22.157.254:45899)
0928 15:50:43.379760 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:43.379767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:43.380708 (+ 941us) server_negotiation.cc:407] Connection header received
0928 15:50:43.382694 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:43.382699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:43.382772 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:43.382872 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:43.384365 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:43.385202 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:43.387155 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:43.387682 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:43.387851 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:43.391863 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:43.391872 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:43.391880 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:43.391931 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:43.395295 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:43.396355 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:43.396360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:43.396362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:43.396573 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:43.397120 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:43.397123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:43.397125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:43.397439 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:43.397553 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:43.398641 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0928 15:50:43.398826 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":229,"threads_started":1}
I20260928 15:50:43.400866 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:43.379790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45899 (local address 127.22.157.194:49581)
0928 15:50:43.380541 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:43.380582 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:43.382420 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:43.383098 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:43.383107 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:43.383632 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:43.384203 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:43.384219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:43.385406 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:43.385410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:43.386983 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:43.386997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:43.387955 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:43.387965 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:43.388224 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:43.389005 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:43.389030 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:43.391594 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:43.395471 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:43.395484 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:43.395507 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:43.396211 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:43.396691 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:43.396699 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:43.396705 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:43.396999 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:43.397595 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:43.397606 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:43.398151 (+ 545us) client_negotiation.cc:769] Sending connection context
0928 15:50:43.398498 (+ 347us) client_negotiation.cc:240] Negotiation successful
0928 15:50:43.399130 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":277,"threads_started":1}
I20260928 15:50:43.402527 25971 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45899
I20260928 15:50:43.402873 25971 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:43.403647 25971 heartbeater.cc:511] Master 127.22.157.254:45899 requested a full tablet report, sending...
I20260928 15:50:43.405233 25646 ts_manager.cc:194] Registered new tserver with Master: 1a8e66fe7fbb4ba786ea71ae1c3a28a7 (127.22.157.194:42573)
I20260928 15:50:43.406142 25646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:49581
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:50:43.436259 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45899
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:43.773901 25978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:43.774266 25978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:43.774446 25978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:43.782368 25978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:43.782545 25978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:43.782737 25978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:50:43.795431 25978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:43.796917 25978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:43.798310 25978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:43.812237 25983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:43.812295 25984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:43.812266 25986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:43.813480 25978 server_base.cc:1034] running on GCE node
I20260928 15:50:43.814222 25978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:43.815217 25978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:43.816422 25978 hybrid_clock.cc:648] HybridClock initialized: now 1790610643816333 us; error 80 us; skew 500 ppm
I20260928 15:50:43.816797 25978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:43 dist-test-slave-1vw0 krb5kdc[25599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610643, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:43.820051 25978 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:50:43.821682 25978 webserver.cc:460] Webserver started at http://127.22.157.195:38863/ using document root <none> and password file <none>
I20260928 15:50:43.822435 25978 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:43.822548 25978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:43.822803 25978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:43.825834 25978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "9da14eb89fe1458eb5c1cfbea8064c03"
format_stamp: "Formatted at 2026-09-28 15:50:43 on dist-test-slave-1vw0"
server_key: "6e9b3e1be1fcbd359c38d93082cf3665"
server_key_iv: "2afb0a48e87fb689c6881898f74c2767"
server_key_version: "encryptionkey@0"
I20260928 15:50:43.826599 25978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "9da14eb89fe1458eb5c1cfbea8064c03"
format_stamp: "Formatted at 2026-09-28 15:50:43 on dist-test-slave-1vw0"
server_key: "6e9b3e1be1fcbd359c38d93082cf3665"
server_key_iv: "2afb0a48e87fb689c6881898f74c2767"
server_key_version: "encryptionkey@0"
I20260928 15:50:43.831820 25978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:50:43.835326 25993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:43.836967 25978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:43.837160 25978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "9da14eb89fe1458eb5c1cfbea8064c03"
format_stamp: "Formatted at 2026-09-28 15:50:43 on dist-test-slave-1vw0"
server_key: "6e9b3e1be1fcbd359c38d93082cf3665"
server_key_iv: "2afb0a48e87fb689c6881898f74c2767"
server_key_version: "encryptionkey@0"
I20260928 15:50:43.837401 25978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:43.848924 25978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:43.853286 25978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:43.853586 25978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:43.854550 25978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:43.855903 25978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:43.856029 25978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:43.856180 25978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:43.856252 25978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:43.880949 25837 heartbeater.cc:503] Master 127.22.157.254:45899 was elected leader, sending a full tablet report...
I20260928 15:50:43.890939 25978 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:39465
I20260928 15:50:43.890976 26107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:39465 every 8 connection(s)
I20260928 15:50:43.892309 25978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:50:43.897392 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 25978
I20260928 15:50:43.897598 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:50:43.898051 23159 external_mini_cluster.cc:1428] Setting key 44b11431cbd6971fb612f31aa8e51c4f
Sep 28 15:50:43 dist-test-slave-1vw0 krb5kdc[25599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610643, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:43.912887 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:43.895237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:46353 (local address 127.22.157.254:45899)
0928 15:50:43.895446 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:43.895451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:43.896690 (+ 1239us) server_negotiation.cc:407] Connection header received
0928 15:50:43.898797 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:43.898803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:43.898919 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:43.899042 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:43.900585 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:43.901542 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:43.903122 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:43.903521 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:43.903654 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:43.907307 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:43.907312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:43.907315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:43.907350 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:43.909801 (+ 2451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:43.910517 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:43.910525 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:43.910533 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:43.910649 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:43.911077 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:43.911083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:43.911086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:43.911366 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:43.911532 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:43.912254 (+ 722us) server_negotiation.cc:299] Negotiation successful
0928 15:50:43.912495 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 15:50:43.913913 26110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:43.895774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45899 (local address 127.22.157.195:46353)
0928 15:50:43.896517 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:43.896562 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:43.898504 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:43.899280 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:43.899288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:43.899826 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:43.900414 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:43.900432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:43.901697 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:43.901701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:43.902971 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:43.902983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:43.903743 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:43.903752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:43.903988 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:43.904669 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:43.904691 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:43.907085 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:43.909947 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:43.909957 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:43.909974 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:43.910389 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:43.910768 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:43.910773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:43.910776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:43.910960 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:43.911503 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:43.911520 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:43.911856 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 15:50:43.912127 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 15:50:43.912678 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":289,"threads_started":1}
I20260928 15:50:43.915298 26108 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45899
I20260928 15:50:43.915647 26108 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:43.916401 26108 heartbeater.cc:511] Master 127.22.157.254:45899 requested a full tablet report, sending...
I20260928 15:50:43.917881 25646 ts_manager.cc:194] Registered new tserver with Master: 9da14eb89fe1458eb5c1cfbea8064c03 (127.22.157.195:39465)
I20260928 15:50:43.918715 25646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:46353
I20260928 15:50:43.925961 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:50:43.958701 25974 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:50:43.956196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.22.157.254:45899)
0928 15:50:43.956441 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:43.956448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:43.956838 (+ 390us) server_negotiation.cc:407] Connection header received
0928 15:50:43.956996 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:43.957000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:43.957102 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:43.957192 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:43.958407 (+ 1215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33056: BlockingRecv error: recv got EOF from 127.0.0.1:33056 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:50:43.961975 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25708
W20260928 15:50:43.964845 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25708: Not authorized: Client connection negotiation failed: client connection to 127.22.157.193:43367: server requires authentication, but client does not have Kerberos credentials available
I20260928 15:50:43.965018 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 25708
I20260928 15:50:43.982046 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25843
W20260928 15:50:43.984881 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25843: Not authorized: Client connection negotiation failed: client connection to 127.22.157.194:42573: server requires authentication, but client does not have Kerberos credentials available
I20260928 15:50:43.985105 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 25843
I20260928 15:50:44.000816 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25978
W20260928 15:50:44.003875 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25978: Not authorized: Client connection negotiation failed: client connection to 127.22.157.195:39465: server requires authentication, but client does not have Kerberos credentials available
I20260928 15:50:44.004040 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 25978
I20260928 15:50:44.020195 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 25612
I20260928 15:50:44.021693 25676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:44.126072 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 25612
2026-09-28T15:50:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3682 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:44 dist-test-slave-1vw0 krb5kdc[26125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:44 dist-test-slave-1vw0 krb5kdc[26125](info): set up 2 sockets
Sep 28 15:50:44 dist-test-slave-1vw0 krb5kdc[26125](info): commencing operation
krb5kdc: starting...
W20260928 15:50:45.273675 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:50:45 dist-test-slave-1vw0 krb5kdc[26125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:50:45.488173 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:36821
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:36821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:45.830145 26138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:45.830489 26138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:45.830669 26138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:45.838248 26138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:45.838390 26138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:45.838516 26138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:45.838606 26138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:45.838697 26138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:45.849131 26138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:36821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:36821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:45.850665 26138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:45.852139 26138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:45.865152 26143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:45.865284 26146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:45.865203 26144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:45.865959 26138 server_base.cc:1034] running on GCE node
I20260928 15:50:45.866806 26138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:45.868319 26138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:45.869598 26138 hybrid_clock.cc:648] HybridClock initialized: now 1790610645869544 us; error 66 us; skew 500 ppm
I20260928 15:50:45.869968 26138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:45 dist-test-slave-1vw0 krb5kdc[26125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610645, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:45.881012 26138 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:50:45.883348 26138 webserver.cc:460] Webserver started at http://127.22.157.254:35059/ using document root <none> and password file <none>
I20260928 15:50:45.884374 26138 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:45.884548 26138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:45.885010 26138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:45.889629 26138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "11569d3fb8234a8791a29ee008d8d78c"
format_stamp: "Formatted at 2026-09-28 15:50:45 on dist-test-slave-1vw0"
server_key: "ce9f9e0f885b74adb04d7e4cf9c510e0"
server_key_iv: "cd1d115fbc4dba03523ac2498b5c5d30"
server_key_version: "encryptionkey@0"
I20260928 15:50:45.890951 26138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "11569d3fb8234a8791a29ee008d8d78c"
format_stamp: "Formatted at 2026-09-28 15:50:45 on dist-test-slave-1vw0"
server_key: "ce9f9e0f885b74adb04d7e4cf9c510e0"
server_key_iv: "cd1d115fbc4dba03523ac2498b5c5d30"
server_key_version: "encryptionkey@0"
I20260928 15:50:45.898990 26138 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260928 15:50:45.904345 26154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:45.906599 26138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:50:45.906829 26138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "11569d3fb8234a8791a29ee008d8d78c"
format_stamp: "Formatted at 2026-09-28 15:50:45 on dist-test-slave-1vw0"
server_key: "ce9f9e0f885b74adb04d7e4cf9c510e0"
server_key_iv: "cd1d115fbc4dba03523ac2498b5c5d30"
server_key_version: "encryptionkey@0"
I20260928 15:50:45.907141 26138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:45.938537 26138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:45.943215 26138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:45.943527 26138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:45.962625 26138 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:36821
I20260928 15:50:45.962656 26205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:36821 every 8 connection(s)
I20260928 15:50:45.964161 26138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:45.968444 26206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:45.972508 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 26138
I20260928 15:50:45.972761 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:50:45.973444 23159 external_mini_cluster.cc:1428] Setting key e4b5b425a2715e879a675466d3ef3aca
I20260928 15:50:45.982393 26206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c: Bootstrap starting.
I20260928 15:50:45.986369 26206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:45.987716 26206 log.cc:824] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:50:45 dist-test-slave-1vw0 krb5kdc[26125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:45.990846 26206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c: No bootstrap required, opened a new log
I20260928 15:50:45.997696 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:45.975488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.22.157.254:36821)
0928 15:50:45.976293 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:45.976304 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:45.976347 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:50:45.978036 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:45.978060 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:45.978498 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:45.978845 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:45.980879 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:45.982212 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:45.983943 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:45.984686 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:45.984894 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:45.989858 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:45.989865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:45.989880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:45.989914 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:45.993218 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:45.994039 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:45.994045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:45.994051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:45.994228 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:45.994632 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:45.994635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:45.994637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:45.995213 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:45.995440 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:45.996001 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 15:50:45.996491 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":373,"threads_started":1}
I20260928 15:50:45.999876 26206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11569d3fb8234a8791a29ee008d8d78c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36821 } }
I20260928 15:50:46.000363 26206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:46.000505 26206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11569d3fb8234a8791a29ee008d8d78c, State: Initialized, Role: FOLLOWER
I20260928 15:50:46.001207 26206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11569d3fb8234a8791a29ee008d8d78c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36821 } }
I20260928 15:50:46.001507 26206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:46.001653 26206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:46.001837 26206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:46.002082 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:50:46.004128 26206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11569d3fb8234a8791a29ee008d8d78c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36821 } }
I20260928 15:50:46.004667 26206 leader_election.cc:302] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11569d3fb8234a8791a29ee008d8d78c; no voters:
I20260928 15:50:46.005360 26206 leader_election.cc:288] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:46.005518 26211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:46.007460 26206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:46.011127 26211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [term 1 LEADER]: Becoming Leader. State: Replica: 11569d3fb8234a8791a29ee008d8d78c, State: Running, Role: LEADER
I20260928 15:50:46.011881 26211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11569d3fb8234a8791a29ee008d8d78c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36821 } }
I20260928 15:50:46.015355 26213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11569d3fb8234a8791a29ee008d8d78c. Latest consensus state: current_term: 1 leader_uuid: "11569d3fb8234a8791a29ee008d8d78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11569d3fb8234a8791a29ee008d8d78c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36821 } } }
I20260928 15:50:46.015348 26212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11569d3fb8234a8791a29ee008d8d78c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11569d3fb8234a8791a29ee008d8d78c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36821 } } }
I20260928 15:50:46.015812 26212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:46.015789 26213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:46.026080 26225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:46.031796 26225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:46.042331 26225 catalog_manager.cc:1347] Generated new cluster ID: 2ae580005aeb430a845da229806bf420
I20260928 15:50:46.042514 26225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:46.052875 26225 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:46.054584 26225 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:46.056002 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:46.046879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.22.157.254:36821)
0928 15:50:46.047197 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:46.047204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:46.047581 (+ 377us) server_negotiation.cc:407] Connection header received
0928 15:50:46.047770 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:46.047776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:46.047865 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:46.047990 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:46.049190 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:46.050353 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:46.051776 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:46.052478 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:46.052617 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:46.054142 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:46.054149 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:46.054155 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:50:46.055053 (+ 898us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:46.055140 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:50:46.055182 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:46.055291 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:46.055465 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 15:50:46.055631 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260928 15:50:46.064937 26225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11569d3fb8234a8791a29ee008d8d78c: Generated new TSK 0
I20260928 15:50:46.065847 26225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:50:46.076543 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26138
I20260928 15:50:46.077826 26201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:46.168357 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 26138
2026-09-28T15:50:46Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2035 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:46 dist-test-slave-1vw0 krb5kdc[26240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:46 dist-test-slave-1vw0 krb5kdc[26240](info): set up 2 sockets
Sep 28 15:50:46 dist-test-slave-1vw0 krb5kdc[26240](info): commencing operation
krb5kdc: starting...
W20260928 15:50:47.349359 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 15:50:47.614948 23159 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.265s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 15:50:47.801259 23159 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.186s user 0.001s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 15:50:47.962980 23159 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.161s user 0.001s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/test-user.keytab.
W20260928 15:50:48.482491 23159 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.519s user 0.001s sys 0.005s
Sep 28 15:50:48 dist-test-slave-1vw0 krb5kdc[26240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:50:48.581163 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:44279
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:44279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:48.920107 26253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:48.920504 26253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:48.920670 26253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:48.928323 26253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:48.928572 26253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:48.928680 26253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:48.928814 26253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:48.928894 26253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:48.939491 26253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:44279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:44279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:48.941011 26253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:48.942345 26253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:48.954854 26259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:48.954877 26261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:48.955304 26258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:48.956312 26253 server_base.cc:1034] running on GCE node
I20260928 15:50:48.957330 26253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:48.958884 26253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:48.960134 26253 hybrid_clock.cc:648] HybridClock initialized: now 1790610648960087 us; error 47 us; skew 500 ppm
I20260928 15:50:48.960512 26253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:48 dist-test-slave-1vw0 krb5kdc[26240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610648, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:48.969522 26253 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:50:48.971230 26253 webserver.cc:460] Webserver started at http://127.22.157.254:39161/ using document root <none> and password file <none>
I20260928 15:50:48.971987 26253 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:48.972144 26253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:48.972470 26253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:48.975659 26253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "0bbf95f8ebda4967a8a97d325217d438"
format_stamp: "Formatted at 2026-09-28 15:50:48 on dist-test-slave-1vw0"
server_key: "ca80e78a5330d6d31d83ac38338a2a61"
server_key_iv: "e9d9b0186bbb82d6ff419de7a58365dd"
server_key_version: "encryptionkey@0"
I20260928 15:50:48.976462 26253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "0bbf95f8ebda4967a8a97d325217d438"
format_stamp: "Formatted at 2026-09-28 15:50:48 on dist-test-slave-1vw0"
server_key: "ca80e78a5330d6d31d83ac38338a2a61"
server_key_iv: "e9d9b0186bbb82d6ff419de7a58365dd"
server_key_version: "encryptionkey@0"
I20260928 15:50:48.981904 26253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:50:48.985630 26268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:48.987051 26253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:50:48.987244 26253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "0bbf95f8ebda4967a8a97d325217d438"
format_stamp: "Formatted at 2026-09-28 15:50:48 on dist-test-slave-1vw0"
server_key: "ca80e78a5330d6d31d83ac38338a2a61"
server_key_iv: "e9d9b0186bbb82d6ff419de7a58365dd"
server_key_version: "encryptionkey@0"
I20260928 15:50:48.987452 26253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:49.007754 26253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:49.012116 26253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:49.012440 26253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:49.031388 26253 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:44279
I20260928 15:50:49.031417 26320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:44279 every 8 connection(s)
I20260928 15:50:49.032886 26253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:49.036882 26321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:49.042212 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 26253
I20260928 15:50:49.042452 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:50:49.043102 23159 external_mini_cluster.cc:1428] Setting key e0aacda0791afcf937a9861219a0004b
I20260928 15:50:49.047951 26321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438: Bootstrap starting.
I20260928 15:50:49.051870 26321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:49.053267 26321 log.cc:824] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:49.056923 26321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438: No bootstrap required, opened a new log
Sep 28 15:50:49 dist-test-slave-1vw0 krb5kdc[26240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:49.065515 26321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbf95f8ebda4967a8a97d325217d438" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44279 } }
I20260928 15:50:49.065901 26321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:49.066047 26321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbf95f8ebda4967a8a97d325217d438, State: Initialized, Role: FOLLOWER
I20260928 15:50:49.066757 26321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbf95f8ebda4967a8a97d325217d438" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44279 } }
I20260928 15:50:49.067013 26321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:49.067163 26321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:49.067377 26321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:49.068012 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:49.045259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.22.157.254:44279)
0928 15:50:49.046161 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:49.046174 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:49.046216 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:50:49.047875 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:49.047898 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:49.048361 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:49.048706 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:49.050783 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:49.052225 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:49.054157 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:49.055169 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:49.055309 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:49.060130 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:49.060139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:49.060158 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:49.060203 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:49.063465 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:49.064193 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:49.064203 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:49.064210 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:49.064381 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:49.064811 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:49.064821 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:49.064823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:49.065378 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:49.065713 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:49.066276 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 15:50:49.066728 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":429,"threads_started":1}
I20260928 15:50:49.069957 26321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbf95f8ebda4967a8a97d325217d438" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44279 } }
I20260928 15:50:49.070473 26321 leader_election.cc:302] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bbf95f8ebda4967a8a97d325217d438; no voters:
I20260928 15:50:49.071131 26321 leader_election.cc:288] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:49.071239 26326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:49.071933 26326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [term 1 LEADER]: Becoming Leader. State: Replica: 0bbf95f8ebda4967a8a97d325217d438, State: Running, Role: LEADER
I20260928 15:50:49.072543 26326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbf95f8ebda4967a8a97d325217d438" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44279 } }
I20260928 15:50:49.073467 26321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:49.075554 26328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bbf95f8ebda4967a8a97d325217d438. Latest consensus state: current_term: 1 leader_uuid: "0bbf95f8ebda4967a8a97d325217d438" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbf95f8ebda4967a8a97d325217d438" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44279 } } }
I20260928 15:50:49.075605 26327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bbf95f8ebda4967a8a97d325217d438" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbf95f8ebda4967a8a97d325217d438" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 44279 } } }
I20260928 15:50:49.075958 26328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:49.075982 26327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:49.078137 26333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:49.083714 26333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:49.093552 26333 catalog_manager.cc:1347] Generated new cluster ID: a1719c64b0eb40b8b074a5d10d98dd35
I20260928 15:50:49.093683 26333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:49.113631 26333 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:49.115013 26333 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:49.124847 26333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bbf95f8ebda4967a8a97d325217d438: Generated new TSK 0
I20260928 15:50:49.125734 26333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:50:49.180802 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:44279
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:49.518517 26347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:49.518818 26347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:49.518960 26347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:49.526533 26347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:49.526648 26347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:49.526824 26347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:50:49.537444 26347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:49.538863 26347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:49.540314 26347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:49.554026 26352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:49.554136 26355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:49.554152 26353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:49.555029 26347 server_base.cc:1034] running on GCE node
I20260928 15:50:49.555747 26347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:49.556833 26347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:49.558103 26347 hybrid_clock.cc:648] HybridClock initialized: now 1790610649558059 us; error 35 us; skew 500 ppm
I20260928 15:50:49.558538 26347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:49 dist-test-slave-1vw0 krb5kdc[26240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610649, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:49.562007 26347 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:50:49.563524 26347 webserver.cc:460] Webserver started at http://127.22.157.193:38833/ using document root <none> and password file <none>
I20260928 15:50:49.564168 26347 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:49.564270 26347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:49.564528 26347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:49.568002 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "8762a302eef045ab9d0d4261ba206fa2"
format_stamp: "Formatted at 2026-09-28 15:50:49 on dist-test-slave-1vw0"
server_key: "4da570d5ddb4cdd1778ebf9633639758"
server_key_iv: "73359a715e3b7bcdeb7408cd181efb53"
server_key_version: "encryptionkey@0"
I20260928 15:50:49.568802 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "8762a302eef045ab9d0d4261ba206fa2"
format_stamp: "Formatted at 2026-09-28 15:50:49 on dist-test-slave-1vw0"
server_key: "4da570d5ddb4cdd1778ebf9633639758"
server_key_iv: "73359a715e3b7bcdeb7408cd181efb53"
server_key_version: "encryptionkey@0"
I20260928 15:50:49.574265 26347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:50:49.578233 26362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:49.579805 26347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:50:49.579979 26347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "8762a302eef045ab9d0d4261ba206fa2"
format_stamp: "Formatted at 2026-09-28 15:50:49 on dist-test-slave-1vw0"
server_key: "4da570d5ddb4cdd1778ebf9633639758"
server_key_iv: "73359a715e3b7bcdeb7408cd181efb53"
server_key_version: "encryptionkey@0"
I20260928 15:50:49.580188 26347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:49.617590 26347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:49.622030 26347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:49.622289 26347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:49.623222 26347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:49.624655 26347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:49.624791 26347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:50:49.624894 26347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:49.624956 26347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:49.662315 26347 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:41031
I20260928 15:50:49.662355 26475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:41031 every 8 connection(s)
I20260928 15:50:49.663724 26347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:50:49.672605 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 26347
I20260928 15:50:49.672837 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:50:49.673674 23159 external_mini_cluster.cc:1428] Setting key 678f5afff79ee7fb5da495bc1949bd72
Sep 28 15:50:49 dist-test-slave-1vw0 krb5kdc[26240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610649, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:49.687357 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:49.666490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:55903 (local address 127.22.157.254:44279)
0928 15:50:49.666969 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:49.666975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:49.668033 (+ 1058us) server_negotiation.cc:407] Connection header received
0928 15:50:49.669967 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:49.669972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:49.670045 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:49.670149 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:49.671941 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:49.673172 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:49.675486 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:49.676039 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:49.676187 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:49.680522 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:49.680531 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:49.680539 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:49.680588 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:49.683707 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:49.684826 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:49.684831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:49.684834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:49.684978 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:49.685549 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:49.685553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:49.685555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:49.685836 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:49.685961 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:49.686927 (+ 966us) server_negotiation.cc:299] Negotiation successful
0928 15:50:49.687105 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":220,"threads_started":1}
I20260928 15:50:49.689059 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:49.667095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44279 (local address 127.22.157.193:55903)
0928 15:50:49.667838 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:49.667878 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:49.669685 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:49.670426 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:49.670436 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:49.671018 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:49.671744 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:49.671765 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:49.673446 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:49.673451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:49.675252 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:49.675270 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:49.676287 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:49.676293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:49.676580 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:49.677488 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:49.677535 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:49.680260 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:49.683896 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:49.683909 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:49.683930 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:49.684645 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:49.685140 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:49.685148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:49.685155 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:49.685412 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:49.685974 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:49.685994 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:49.686495 (+ 501us) client_negotiation.cc:769] Sending connection context
0928 15:50:49.686817 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 15:50:49.687467 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":319,"threads_started":1}
I20260928 15:50:49.690932 26476 heartbeater.cc:347] Connected to a master server at 127.22.157.254:44279
I20260928 15:50:49.691386 26476 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:49.692394 26476 heartbeater.cc:511] Master 127.22.157.254:44279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
I20260928 15:50:49.694829 26285 ts_manager.cc:194] Registered new tserver with Master: 8762a302eef045ab9d0d4261ba206fa2 (127.22.157.193:41031)
I20260928 15:50:49.696419 26285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:55903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:50:49.724861 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:44279
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:50.055866 26482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:50.056203 26482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:50.056382 26482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:50.063961 26482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:50.064126 26482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:50.064316 26482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:50:50.074891 26482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:50.076267 26482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:50.077762 26482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:50.091233 26487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:50.091233 26490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:50.091362 26488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:50.092561 26482 server_base.cc:1034] running on GCE node
I20260928 15:50:50.093343 26482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:50.094471 26482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:50.095734 26482 hybrid_clock.cc:648] HybridClock initialized: now 1790610650095682 us; error 61 us; skew 500 ppm
I20260928 15:50:50.096122 26482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:50 dist-test-slave-1vw0 krb5kdc[26240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610650, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:50.099602 26482 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:50:50.101166 26482 webserver.cc:460] Webserver started at http://127.22.157.194:41191/ using document root <none> and password file <none>
I20260928 15:50:50.101914 26482 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:50.102062 26482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:50.102380 26482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:50.105623 26482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "48b459e90d784d7ebeb7f083aa0d1dd5"
format_stamp: "Formatted at 2026-09-28 15:50:50 on dist-test-slave-1vw0"
server_key: "77e403cf8ac3491266d33fbb238b2c05"
server_key_iv: "aea2cea1e4a6b7ed69e75722e45656f9"
server_key_version: "encryptionkey@0"
I20260928 15:50:50.106464 26482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "48b459e90d784d7ebeb7f083aa0d1dd5"
format_stamp: "Formatted at 2026-09-28 15:50:50 on dist-test-slave-1vw0"
server_key: "77e403cf8ac3491266d33fbb238b2c05"
server_key_iv: "aea2cea1e4a6b7ed69e75722e45656f9"
server_key_version: "encryptionkey@0"
I20260928 15:50:50.111788 26482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:50:50.115465 26497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:50.116937 26482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:50.117120 26482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "48b459e90d784d7ebeb7f083aa0d1dd5"
format_stamp: "Formatted at 2026-09-28 15:50:50 on dist-test-slave-1vw0"
server_key: "77e403cf8ac3491266d33fbb238b2c05"
server_key_iv: "aea2cea1e4a6b7ed69e75722e45656f9"
server_key_version: "encryptionkey@0"
I20260928 15:50:50.117326 26482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:50.134148 26482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:50.138485 26482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:50.138792 26482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:50.139804 26482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:50.141323 26482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:50.141597 26482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:50.141770 26482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:50.141852 26482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:50.176318 26482 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:34283
I20260928 15:50:50.176362 26610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:34283 every 8 connection(s)
I20260928 15:50:50.177775 26482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:50:50.185760 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 26482
I20260928 15:50:50.185948 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:50:50.186412 23159 external_mini_cluster.cc:1428] Setting key 5dce29e5a0e963384cf9159109a1062f
Sep 28 15:50:50 dist-test-slave-1vw0 krb5kdc[26240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610650, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:50.199928 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:50.180354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:59465 (local address 127.22.157.254:44279)
0928 15:50:50.180613 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:50.180620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:50.181871 (+ 1251us) server_negotiation.cc:407] Connection header received
0928 15:50:50.183684 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:50.183688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:50.183759 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:50.183897 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:50.185552 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.186488 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.188533 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.189136 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.189273 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:50.193462 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:50.193471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:50.193478 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:50.193532 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:50.196669 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.197518 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.197524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.197528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.197666 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.198129 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.198134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.198138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.198415 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:50.198586 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:50.199324 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 15:50:50.199585 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 15:50:50.201040 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:50.180963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44279 (local address 127.22.157.194:59465)
0928 15:50:50.181709 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:50.181752 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:50.183369 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:50.184121 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:50.184129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:50.184687 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:50.185346 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:50.185371 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.186687 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.186692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:50.188388 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:50.188399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.189355 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.189360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:50.189613 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:50.190351 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:50.190386 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:50.193222 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:50.196909 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:50.196918 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:50.196936 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:50.197376 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:50.197784 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:50.197789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:50.197793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:50.198006 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:50.198552 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:50.198560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:50.198931 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 15:50:50.199218 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 15:50:50.199783 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":300,"threads_started":1}
I20260928 15:50:50.202679 26611 heartbeater.cc:347] Connected to a master server at 127.22.157.254:44279
I20260928 15:50:50.203006 26611 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:50.203953 26611 heartbeater.cc:511] Master 127.22.157.254:44279 requested a full tablet report, sending...
I20260928 15:50:50.205586 26285 ts_manager.cc:194] Registered new tserver with Master: 48b459e90d784d7ebeb7f083aa0d1dd5 (127.22.157.194:34283)
I20260928 15:50:50.206420 26285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:59465
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:50:50.237939 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:44279
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:33479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:50.564150 26616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:50.564515 26616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:50.564702 26616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:50.572201 26616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:50.572384 26616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:50.572582 26616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:50:50.582825 26616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:33479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:50.584385 26616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:50.585940 26616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:50.600033 26624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:50.600088 26622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:50.600081 26621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:50.600762 26616 server_base.cc:1034] running on GCE node
I20260928 15:50:50.601460 26616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:50.602560 26616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:50.603837 26616 hybrid_clock.cc:648] HybridClock initialized: now 1790610650603746 us; error 62 us; skew 500 ppm
I20260928 15:50:50.604202 26616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:50 dist-test-slave-1vw0 krb5kdc[26240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610650, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:50.607712 26616 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:50:50.609272 26616 webserver.cc:460] Webserver started at http://127.22.157.195:36937/ using document root <none> and password file <none>
I20260928 15:50:50.609998 26616 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:50.610145 26616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:50.610466 26616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:50.613540 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "03fbd1be96e84bd99c5e005a969da4e3"
format_stamp: "Formatted at 2026-09-28 15:50:50 on dist-test-slave-1vw0"
server_key: "72785ee702678203e19468172c620831"
server_key_iv: "b641396735a517c180f54f0e6e74432f"
server_key_version: "encryptionkey@0"
I20260928 15:50:50.614369 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "03fbd1be96e84bd99c5e005a969da4e3"
format_stamp: "Formatted at 2026-09-28 15:50:50 on dist-test-slave-1vw0"
server_key: "72785ee702678203e19468172c620831"
server_key_iv: "b641396735a517c180f54f0e6e74432f"
server_key_version: "encryptionkey@0"
I20260928 15:50:50.620038 26616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:50:50.623728 26631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:50.625293 26616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:50.625520 26616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "03fbd1be96e84bd99c5e005a969da4e3"
format_stamp: "Formatted at 2026-09-28 15:50:50 on dist-test-slave-1vw0"
server_key: "72785ee702678203e19468172c620831"
server_key_iv: "b641396735a517c180f54f0e6e74432f"
server_key_version: "encryptionkey@0"
I20260928 15:50:50.625737 26616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:50.638845 26616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:50.643386 26616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:50.643711 26616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:50.644691 26616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:50.646063 26616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:50.646168 26616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:50.646312 26616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:50.646384 26616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:50.680851 26616 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:38757
I20260928 15:50:50.680887 26743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:38757 every 8 connection(s)
I20260928 15:50:50.682180 26616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:50:50.688611 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 26616
I20260928 15:50:50.688827 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:50:50.689299 23159 external_mini_cluster.cc:1428] Setting key 585274cd284da829cbbe423d0648221b
Sep 28 15:50:50 dist-test-slave-1vw0 krb5kdc[26240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610650, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:50.700520 26476 heartbeater.cc:503] Master 127.22.157.254:44279 was elected leader, sending a full tablet report...
I20260928 15:50:50.703711 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:50.684951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:33873 (local address 127.22.157.254:44279)
0928 15:50:50.685183 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:50.685189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:50.686390 (+ 1201us) server_negotiation.cc:407] Connection header received
0928 15:50:50.688301 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:50.688306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:50.688426 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:50.688542 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:50.690400 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.691469 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.693273 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.693687 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.693859 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:50.697432 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:50.697437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:50.697440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:50.697474 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:50.700069 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.700949 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.700956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.700964 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.701134 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.701718 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.701725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.701731 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.702064 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:50.702247 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:50.703085 (+ 838us) server_negotiation.cc:299] Negotiation successful
0928 15:50:50.703328 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 15:50:50.704964 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:50.685473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:44279 (local address 127.22.157.195:33873)
0928 15:50:50.686225 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:50.686265 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:50.688009 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:50.688828 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:50.688847 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:50.689485 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:50.690227 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:50.690249 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.691630 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.691634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:50.693120 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:50.693133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.693928 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.693938 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:50.694202 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:50.694864 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:50.694887 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:50.697201 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:50.700239 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:50.700248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:50.700295 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:50.700823 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:50.701252 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:50.701256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:50.701262 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:50.701565 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:50.702250 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:50.702258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:50.702660 (+ 402us) client_negotiation.cc:769] Sending connection context
0928 15:50:50.702943 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 15:50:50.703546 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":280,"threads_started":1}
I20260928 15:50:50.706234 26744 heartbeater.cc:347] Connected to a master server at 127.22.157.254:44279
I20260928 15:50:50.706570 26744 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:50.707340 26744 heartbeater.cc:511] Master 127.22.157.254:44279 requested a full tablet report, sending...
I20260928 15:50:50.708693 26285 ts_manager.cc:194] Registered new tserver with Master: 03fbd1be96e84bd99c5e005a969da4e3 (127.22.157.195:38757)
I20260928 15:50:50.709540 26285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:33873
I20260928 15:50:50.717070 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:50:50 dist-test-slave-1vw0 krb5kdc[26240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 15:50:50 dist-test-slave-1vw0 krb5kdc[26240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:50.769284 26478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:50.754327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.22.157.254:44279)
0928 15:50:50.754585 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:50.754594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:50.755015 (+ 421us) server_negotiation.cc:407] Connection header received
0928 15:50:50.755236 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:50.755240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:50.755307 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:50.755434 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:50.756927 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.757655 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.758941 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.759359 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.759506 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:50.763650 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:50.763656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:50.763659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:50.763699 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:50.766270 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.766970 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.766975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.766981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.767121 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.767706 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.767719 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.767722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.768094 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:50.768380 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:50.768706 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 15:50:50.768962 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260928 15:50:50.769851 26285 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45722
I20260928 15:50:50.771505 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26347
Sep 28 15:50:50 dist-test-slave-1vw0 krb5kdc[26240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:50:50.788584 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:50.772681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.22.157.193:41031)
0928 15:50:50.773381 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:50.773388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:50.773414 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:50:50.773471 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:50.773477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:50.773772 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:50.773905 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:50.775209 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.776222 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.777523 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.778086 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.778240 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:50.782496 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:50.782511 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:50.782526 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:50.782593 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:50.785840 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.786629 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.786636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.786638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.786766 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.787162 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.787165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.787167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.787504 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:50.787782 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:50.788061 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 15:50:50.788265 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":371,"threads_started":1}
W20260928 15:50:50.789961 26471 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50870
W20260928 15:50:50.791215 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26347: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:50.791466 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 26347
I20260928 15:50:50.808588 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26482
Sep 28 15:50:50 dist-test-slave-1vw0 krb5kdc[26240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:50:50.825520 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:50.809703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.22.157.194:34283)
0928 15:50:50.810274 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:50.810282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:50.810310 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:50:50.810371 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:50.810377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:50.810615 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:50.810754 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:50.812133 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.813387 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.814694 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.815284 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.815392 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:50.819708 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:50.819720 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:50.819735 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:50.819774 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:50.822937 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.823555 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.823562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.823564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.823725 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.824070 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.824074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.824076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.824409 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:50.824659 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:50.824970 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 15:50:50.825173 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":288,"threads_started":1}
W20260928 15:50:50.826797 26606 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50158
W20260928 15:50:50.827956 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26482: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:50.828157 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 26482
I20260928 15:50:50.844040 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26616
Sep 28 15:50:50 dist-test-slave-1vw0 krb5kdc[26240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:50:50.861940 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:50.845165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.22.157.195:38757)
0928 15:50:50.845841 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:50.845848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:50.845876 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:50:50.845935 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:50.845941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:50.846173 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:50.846296 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:50.847691 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.848880 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.850222 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:50.850845 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:50.850984 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:50.855821 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:50.855835 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:50.855851 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:50.855913 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:50.858987 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.859741 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.859749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.859751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.859878 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:50.860263 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:50.860267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:50.860270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:50.860651 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:50.861077 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:50.861423 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 15:50:50.861644 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":377,"threads_started":1}
W20260928 15:50:50.863298 26739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51764
W20260928 15:50:50.864526 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26616: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:50:50.864704 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 26616
I20260928 15:50:50.880466 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26253
I20260928 15:50:50.881846 26316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:50.986569 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 26253
2026-09-28T15:50:51Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4824 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:51 dist-test-slave-1vw0 krb5kdc[26762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:51 dist-test-slave-1vw0 krb5kdc[26762](info): set up 2 sockets
Sep 28 15:50:51 dist-test-slave-1vw0 krb5kdc[26762](info): commencing operation
krb5kdc: starting...
W20260928 15:50:52.114307 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:50:52 dist-test-slave-1vw0 krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:50:52.333014 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:36641
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:36641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:52.673749 26775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:52.674132 26775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:52.674328 26775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:52.681942 26775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:52.682071 26775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:52.682169 26775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:52.682266 26775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:52.682350 26775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:52.692603 26775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:36641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:36641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:52.694126 26775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:52.695498 26775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:52.709492 26781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:52.709441 26784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:52.709407 26782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:52.710376 26775 server_base.cc:1034] running on GCE node
I20260928 15:50:52.711252 26775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:52.713064 26775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:52.714336 26775 hybrid_clock.cc:648] HybridClock initialized: now 1790610652714275 us; error 51 us; skew 500 ppm
I20260928 15:50:52.714733 26775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:52 dist-test-slave-1vw0 krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610652, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:52.718180 26775 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:50:52.720069 26775 webserver.cc:460] Webserver started at http://127.22.157.254:41839/ using document root <none> and password file <none>
I20260928 15:50:52.721225 26775 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:52.721398 26775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:52.721745 26775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:52.725010 26775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "5ec6618061c74faa94e324c89591dd92"
format_stamp: "Formatted at 2026-09-28 15:50:52 on dist-test-slave-1vw0"
server_key: "2899eb6521b8bc6eaf9794b035b815f0"
server_key_iv: "d7d8aeaf7b6658f8329271d81caeed6f"
server_key_version: "encryptionkey@0"
I20260928 15:50:52.725824 26775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "5ec6618061c74faa94e324c89591dd92"
format_stamp: "Formatted at 2026-09-28 15:50:52 on dist-test-slave-1vw0"
server_key: "2899eb6521b8bc6eaf9794b035b815f0"
server_key_iv: "d7d8aeaf7b6658f8329271d81caeed6f"
server_key_version: "encryptionkey@0"
I20260928 15:50:52.732080 26775 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 15:50:52.736155 26791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:52.737632 26775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:52.737836 26775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "5ec6618061c74faa94e324c89591dd92"
format_stamp: "Formatted at 2026-09-28 15:50:52 on dist-test-slave-1vw0"
server_key: "2899eb6521b8bc6eaf9794b035b815f0"
server_key_iv: "d7d8aeaf7b6658f8329271d81caeed6f"
server_key_version: "encryptionkey@0"
I20260928 15:50:52.738062 26775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:52.762951 26775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:52.767318 26775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:52.767630 26775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:52.787416 26775 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:36641
I20260928 15:50:52.787420 26842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:36641 every 8 connection(s)
I20260928 15:50:52.789274 26775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:52.793610 26843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:52.795171 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 26775
I20260928 15:50:52.795416 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:50:52.796255 23159 external_mini_cluster.cc:1428] Setting key 02b3c14f0b92964485bdbe9a1f923fda
I20260928 15:50:52.807878 26843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92: Bootstrap starting.
I20260928 15:50:52.811496 26843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92: Neither blocks nor log segments found. Creating new log.
Sep 28 15:50:52 dist-test-slave-1vw0 krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:52.813035 26843 log.cc:824] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:52.817459 26843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92: No bootstrap required, opened a new log
I20260928 15:50:52.823035 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:52.798715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.22.157.254:36641)
0928 15:50:52.799616 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:52.799632 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:52.799678 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:50:52.801392 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:52.801420 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:52.801895 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:52.802339 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:52.804230 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:52.805741 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:52.807091 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:52.807823 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:52.808061 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:52.813127 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:52.813139 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:52.813164 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:52.813223 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:52.817690 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:52.818672 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:52.818682 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:52.818691 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:52.818949 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:52.819329 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:52.819339 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:52.819343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:52.820182 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:52.820521 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:52.820977 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 15:50:52.821511 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":419,"threads_started":1}
I20260928 15:50:52.827960 26843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec6618061c74faa94e324c89591dd92" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36641 } }
I20260928 15:50:52.828349 26843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:52.828429 26843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec6618061c74faa94e324c89591dd92, State: Initialized, Role: FOLLOWER
I20260928 15:50:52.829042 26843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec6618061c74faa94e324c89591dd92" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36641 } }
I20260928 15:50:52.829252 26843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:52.829424 26843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:52.829620 26843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:52.831950 26843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec6618061c74faa94e324c89591dd92" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36641 } }
I20260928 15:50:52.832499 26843 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec6618061c74faa94e324c89591dd92; no voters:
I20260928 15:50:52.833288 26843 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:52.833400 26848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:52.834236 26848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec6618061c74faa94e324c89591dd92, State: Running, Role: LEADER
I20260928 15:50:52.834870 26848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec6618061c74faa94e324c89591dd92" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36641 } }
I20260928 15:50:52.835654 26843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:52.837874 26849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ec6618061c74faa94e324c89591dd92. Latest consensus state: current_term: 1 leader_uuid: "5ec6618061c74faa94e324c89591dd92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec6618061c74faa94e324c89591dd92" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36641 } } }
I20260928 15:50:52.838361 26849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:52.842916 26850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ec6618061c74faa94e324c89591dd92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec6618061c74faa94e324c89591dd92" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36641 } } }
I20260928 15:50:52.844146 26850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:52.845278 26856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:52.852044 26856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:52.863111 26856 catalog_manager.cc:1347] Generated new cluster ID: 2e6bd990c05947fe9f4614d32289ee34
I20260928 15:50:52.863269 26856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:52.875275 26856 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:52.876917 26856 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:52.884164 26856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ec6618061c74faa94e324c89591dd92: Generated new TSK 0
I20260928 15:50:52.885342 26856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:50:52.954771 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36641
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:53.311942 26870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:53.312258 26870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:53.312415 26870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:53.320343 26870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:53.320477 26870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:53.320678 26870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:50:53.331391 26870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:53.333011 26870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:53.334466 26870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:53.351452 26876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:53.351575 26875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:53.351480 26878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:53.358110 26870 server_base.cc:1034] running on GCE node
I20260928 15:50:53.358857 26870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:53.359896 26870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:53.361151 26870 hybrid_clock.cc:648] HybridClock initialized: now 1790610653361096 us; error 57 us; skew 500 ppm
I20260928 15:50:53.361574 26870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:53 dist-test-slave-1vw0 krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610653, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:53.365391 26870 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:50:53.367190 26870 webserver.cc:460] Webserver started at http://127.22.157.193:45335/ using document root <none> and password file <none>
I20260928 15:50:53.367997 26870 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:53.368125 26870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:53.368407 26870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:53.371589 26870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "3acc2eaf284f43b798c59d6f96700407"
format_stamp: "Formatted at 2026-09-28 15:50:53 on dist-test-slave-1vw0"
server_key: "ce91fd46ca9b169af2c61b65762fec3e"
server_key_iv: "9ace911c627b96f5afc369dcadbffee1"
server_key_version: "encryptionkey@0"
I20260928 15:50:53.372416 26870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "3acc2eaf284f43b798c59d6f96700407"
format_stamp: "Formatted at 2026-09-28 15:50:53 on dist-test-slave-1vw0"
server_key: "ce91fd46ca9b169af2c61b65762fec3e"
server_key_iv: "9ace911c627b96f5afc369dcadbffee1"
server_key_version: "encryptionkey@0"
I20260928 15:50:53.378216 26870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:50:53.382181 26885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:53.384199 26870 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:50:53.384456 26870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "3acc2eaf284f43b798c59d6f96700407"
format_stamp: "Formatted at 2026-09-28 15:50:53 on dist-test-slave-1vw0"
server_key: "ce91fd46ca9b169af2c61b65762fec3e"
server_key_iv: "9ace911c627b96f5afc369dcadbffee1"
server_key_version: "encryptionkey@0"
I20260928 15:50:53.384804 26870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:53.406111 26870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:53.411968 26870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:53.412315 26870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:53.413625 26870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:53.415222 26870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:53.415341 26870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:53.415506 26870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:53.415585 26870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:53.452607 26870 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:38963
I20260928 15:50:53.452649 26998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:38963 every 8 connection(s)
I20260928 15:50:53.454360 26870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:50:53.460115 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 26870
I20260928 15:50:53.460373 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:50:53.461174 23159 external_mini_cluster.cc:1428] Setting key e4bbd76ce0b13cb0d8ec314f5c05c614
Sep 28 15:50:53 dist-test-slave-1vw0 krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610653, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:53.480674 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:53.457576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:55663 (local address 127.22.157.254:36641)
0928 15:50:53.458130 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:53.458135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:53.459376 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 15:50:53.461697 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:53.461703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:53.461820 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:53.461970 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:53.464051 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:53.464927 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:53.467161 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:53.467608 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:53.467790 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:53.472164 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:53.472175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:53.472183 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:53.472236 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:53.476413 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:53.477471 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:53.477480 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:53.477486 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:53.477654 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:53.478236 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:53.478244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:53.478251 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:53.478691 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:53.478865 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:53.479798 (+ 933us) server_negotiation.cc:299] Negotiation successful
0928 15:50:53.480118 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":292,"threads_started":1}
I20260928 15:50:53.481580 27002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:53.458303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:36641 (local address 127.22.157.193:55663)
0928 15:50:53.459157 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:53.459199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:53.461354 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:53.462274 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:53.462285 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:53.463175 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:53.463839 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:53.463861 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:53.465137 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:53.465142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:53.466872 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:53.466895 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:53.467916 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:53.467924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:53.468227 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:53.469140 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:53.469171 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:53.471833 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:53.476642 (+ 4809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:53.476652 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:53.476670 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:53.477289 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:53.477827 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:53.477832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:53.477834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:53.478081 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:53.478839 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:53.478846 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:53.479240 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 15:50:53.479614 (+ 374us) client_negotiation.cc:240] Negotiation successful
0928 15:50:53.480136 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":323,"threads_started":1}
I20260928 15:50:53.483487 26999 heartbeater.cc:347] Connected to a master server at 127.22.157.254:36641
I20260928 15:50:53.483847 26999 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
I20260928 15:50:53.484686 26999 heartbeater.cc:511] Master 127.22.157.254:36641 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
I20260928 15:50:53.487228 26808 ts_manager.cc:194] Registered new tserver with Master: 3acc2eaf284f43b798c59d6f96700407 (127.22.157.193:38963)
I20260928 15:50:53.488986 26808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:55663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:50:53.516443 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36641
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:53.873899 27005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:53.874279 27005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:53.874476 27005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:53.882789 27005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:53.882930 27005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:53.883160 27005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:50:53.893989 27005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:53.895541 27005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:53.897125 27005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:53.911804 27010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:53.911917 27013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:53.911896 27011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:53.912825 27005 server_base.cc:1034] running on GCE node
I20260928 15:50:53.913509 27005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:53.914654 27005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:53.915963 27005 hybrid_clock.cc:648] HybridClock initialized: now 1790610653915873 us; error 68 us; skew 500 ppm
I20260928 15:50:53.916332 27005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:53 dist-test-slave-1vw0 krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610653, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:53.919893 27005 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:50:53.921705 27005 webserver.cc:460] Webserver started at http://127.22.157.194:33077/ using document root <none> and password file <none>
I20260928 15:50:53.922447 27005 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:53.922592 27005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:53.922947 27005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:53.926105 27005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "4132ff88552744d7a77d5a1e1048b360"
format_stamp: "Formatted at 2026-09-28 15:50:53 on dist-test-slave-1vw0"
server_key: "b37a2ecda57f2f8e0e6527802cf0e2d4"
server_key_iv: "b473d7c90a262dceb6f8c75b8add0551"
server_key_version: "encryptionkey@0"
I20260928 15:50:53.926987 27005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "4132ff88552744d7a77d5a1e1048b360"
format_stamp: "Formatted at 2026-09-28 15:50:53 on dist-test-slave-1vw0"
server_key: "b37a2ecda57f2f8e0e6527802cf0e2d4"
server_key_iv: "b473d7c90a262dceb6f8c75b8add0551"
server_key_version: "encryptionkey@0"
I20260928 15:50:53.933044 27005 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 15:50:53.937484 27020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:53.939260 27005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 15:50:53.939592 27005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "4132ff88552744d7a77d5a1e1048b360"
format_stamp: "Formatted at 2026-09-28 15:50:53 on dist-test-slave-1vw0"
server_key: "b37a2ecda57f2f8e0e6527802cf0e2d4"
server_key_iv: "b473d7c90a262dceb6f8c75b8add0551"
server_key_version: "encryptionkey@0"
I20260928 15:50:53.939886 27005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:53.957257 27005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:53.962466 27005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:53.962836 27005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:53.964026 27005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:53.965626 27005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:53.965701 27005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:53.965843 27005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:53.965915 27005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:54.002286 27005 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:34753
I20260928 15:50:54.002336 27133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:34753 every 8 connection(s)
I20260928 15:50:54.003695 27005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:50:54.009925 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 27005
I20260928 15:50:54.010143 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:50:54.010717 23159 external_mini_cluster.cc:1428] Setting key 995004e78f5505a4244f0daa06dac8fe
Sep 28 15:50:54 dist-test-slave-1vw0 krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610653, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
I20260928 15:50:54.040345 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:54.006671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:33937 (local address 127.22.157.254:36641)
0928 15:50:54.007197 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:54.007202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:54.008369 (+ 1167us) server_negotiation.cc:407] Connection header received
0928 15:50:54.011011 (+ 2642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:54.011016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:54.011114 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:54.011226 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:54.013332 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.014194 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.017211 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.017727 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.017986 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:54.031809 (+ 13823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:54.031820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:54.031826 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:54.031900 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:54.035399 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.037098 (+ 1699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.037105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.037109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.037312 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.037922 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.037932 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.037938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.038400 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:54.038569 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:54.039430 (+ 861us) server_negotiation.cc:299] Negotiation successful
0928 15:50:54.039744 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":288,"threads_started":1}
I20260928 15:50:54.041070 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:54.007319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:36641 (local address 127.22.157.194:33937)
0928 15:50:54.008151 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:54.008198 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:54.010522 (+ 2324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:54.011554 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:54.011563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:54.012248 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:54.013101 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:54.013122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.014411 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.014420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:54.016955 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:54.016983 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.018108 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.018118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:54.018533 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:54.028291 (+ 9758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:54.028338 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:54.031366 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:54.035766 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:54.035777 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:54.035794 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:54.036898 (+ 1104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:54.037502 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:54.037506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:54.037509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:54.037735 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:54.038553 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:54.038561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:54.038969 (+ 408us) client_negotiation.cc:769] Sending connection context
0928 15:50:54.039228 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 15:50:54.039745 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":346,"threads_started":1}
I20260928 15:50:54.042893 27134 heartbeater.cc:347] Connected to a master server at 127.22.157.254:36641
I20260928 15:50:54.043263 27134 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:54.044154 27134 heartbeater.cc:511] Master 127.22.157.254:36641 requested a full tablet report, sending...
I20260928 15:50:54.046118 26808 ts_manager.cc:194] Registered new tserver with Master: 4132ff88552744d7a77d5a1e1048b360 (127.22.157.194:34753)
I20260928 15:50:54.047211 26808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:33937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:50:54.069711 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36641
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:54.416141 27140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:54.416528 27140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:54.416761 27140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:54.425096 27140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:54.425261 27140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:54.425441 27140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:50:54.436570 27140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:54.438611 27140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:54.440256 27140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:54.457321 27148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:54.457321 27146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:54.458717 27145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:54.459142 27140 server_base.cc:1034] running on GCE node
I20260928 15:50:54.459980 27140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:54.461170 27140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:54.462452 27140 hybrid_clock.cc:648] HybridClock initialized: now 1790610654462413 us; error 39 us; skew 500 ppm
I20260928 15:50:54.462915 27140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:54 dist-test-slave-1vw0 krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610654, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:54.467093 27140 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:50:54.469185 27140 webserver.cc:460] Webserver started at http://127.22.157.195:32845/ using document root <none> and password file <none>
I20260928 15:50:54.470137 27140 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:54.470332 27140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:54.470762 27140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:54.474813 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "f939fea59bf9452087b06165ab72012b"
format_stamp: "Formatted at 2026-09-28 15:50:54 on dist-test-slave-1vw0"
server_key: "fc9229be86d2c33edcfda18fe3075ec0"
server_key_iv: "7024b0f1480d7b5002b086dd7e4829ee"
server_key_version: "encryptionkey@0"
I20260928 15:50:54.475833 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "f939fea59bf9452087b06165ab72012b"
format_stamp: "Formatted at 2026-09-28 15:50:54 on dist-test-slave-1vw0"
server_key: "fc9229be86d2c33edcfda18fe3075ec0"
server_key_iv: "7024b0f1480d7b5002b086dd7e4829ee"
server_key_version: "encryptionkey@0"
I20260928 15:50:54.482231 27140 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 15:50:54.486696 27155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:54.488411 27140 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260928 15:50:54.488798 27140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "f939fea59bf9452087b06165ab72012b"
format_stamp: "Formatted at 2026-09-28 15:50:54 on dist-test-slave-1vw0"
server_key: "fc9229be86d2c33edcfda18fe3075ec0"
server_key_iv: "7024b0f1480d7b5002b086dd7e4829ee"
server_key_version: "encryptionkey@0"
I20260928 15:50:54.489086 27140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:54.492650 26999 heartbeater.cc:503] Master 127.22.157.254:36641 was elected leader, sending a full tablet report...
I20260928 15:50:54.527194 27140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:54.532869 27140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:54.533161 27140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:54.534425 27140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:54.536331 27140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:54.536429 27140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:54.536602 27140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:54.536679 27140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:54.576682 27140 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:35667
I20260928 15:50:54.576781 27268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:35667 every 8 connection(s)
I20260928 15:50:54.578207 27140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:50:54.584874 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 27140
I20260928 15:50:54.585108 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:50:54.585744 23159 external_mini_cluster.cc:1428] Setting key d6b80394acf8e914f6d78ba5c92d74ea
Sep 28 15:50:54 dist-test-slave-1vw0 krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610654, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:54.604782 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:54.581287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:45217 (local address 127.22.157.254:36641)
0928 15:50:54.581874 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:54.581878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:54.582787 (+ 909us) server_negotiation.cc:407] Connection header received
0928 15:50:54.585102 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:54.585110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:54.585257 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:54.585362 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:54.587844 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.589318 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.591551 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.592010 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.592133 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:54.596985 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:54.596991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:54.596994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:54.597050 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:54.600410 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.601800 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.601805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.601807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.602033 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.602723 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.602728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.602731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.603196 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:54.603329 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:54.604196 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 15:50:54.604407 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":356,"threads_started":1}
I20260928 15:50:54.605986 27272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:54.581786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:36641 (local address 127.22.157.195:45217)
0928 15:50:54.582573 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:54.582618 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:54.584641 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:54.585703 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:54.585717 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:54.586605 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:54.587580 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:54.587614 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.589589 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.589596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:54.591355 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:54.591375 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.592311 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.592317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:54.592587 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:54.593598 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:54.593632 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:54.596592 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:54.600707 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:54.600718 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:54.600796 (+ 78us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:54.601635 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:54.602218 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:54.602223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:54.602226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:54.602560 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:54.603342 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:54.603349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:54.603809 (+ 460us) client_negotiation.cc:769] Sending connection context
0928 15:50:54.604069 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 15:50:54.604548 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":314,"threads_started":1}
I20260928 15:50:54.608117 27269 heartbeater.cc:347] Connected to a master server at 127.22.157.254:36641
I20260928 15:50:54.608515 27269 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:54.609494 27269 heartbeater.cc:511] Master 127.22.157.254:36641 requested a full tablet report, sending...
I20260928 15:50:54.611635 26808 ts_manager.cc:194] Registered new tserver with Master: f939fea59bf9452087b06165ab72012b (127.22.157.195:35667)
I20260928 15:50:54.612576 26808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:45217
I20260928 15:50:54.615895 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:50:54 dist-test-slave-1vw0 krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:50:54 dist-test-slave-1vw0 krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:50:54.672706 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:54.655323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.22.157.193:38963)
0928 15:50:54.656023 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:54.656030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:54.656061 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:50:54.656124 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:54.656130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:54.656352 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:54.656489 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:54.658218 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.659327 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.660707 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.661418 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.661531 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:54.666088 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:54.666104 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:54.666121 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:54.666164 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:54.669568 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.670409 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.670418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.670421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.670586 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.671057 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.671061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.671063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.671455 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:54.671725 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:54.672112 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 15:50:54.672351 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":374,"threads_started":1}
Sep 28 15:50:54 dist-test-slave-1vw0 krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:54.695595 27271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:54.678032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.22.157.254:36641)
0928 15:50:54.678280 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:54.678288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:54.678426 (+ 138us) server_negotiation.cc:407] Connection header received
0928 15:50:54.678684 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:54.678689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:54.678760 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:54.678850 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:54.680372 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.681307 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.682669 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.683177 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.683294 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:54.688566 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:54.688574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:54.688578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:54.688629 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:54.691857 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.692904 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.692911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.692913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.693146 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.693750 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.693755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.693757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.694257 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:54.694602 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:54.694970 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 15:50:54.695208 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 15:50:54.696252 26808 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55968
I20260928 15:50:54.697613 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26870
I20260928 15:50:54.712880 27277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:54.698824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.22.157.193:38963)
0928 15:50:54.699146 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:54.699153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:54.699368 (+ 215us) server_negotiation.cc:407] Connection header received
0928 15:50:54.699539 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:54.699544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:54.699650 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:54.699739 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:54.701187 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.701925 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.703507 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.703981 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.704105 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:54.706954 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:54.706960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:54.706964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:54.707007 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:54.709878 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.710712 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.710718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.710722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.710876 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.711315 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.711321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.711324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.711626 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:54.711904 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:54.712242 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 15:50:54.712494 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260928 15:50:54.713585 26994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:54.801126 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 26870
I20260928 15:50:54.818696 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 27005
Sep 28 15:50:54 dist-test-slave-1vw0 krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:50:54.840359 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:54.820066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.22.157.194:34753)
0928 15:50:54.820953 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:54.820965 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:54.821002 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:50:54.821082 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:54.821091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:54.821457 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:54.821656 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:54.823402 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.824848 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.826323 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.827043 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.827167 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:54.832262 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:54.832281 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:54.832298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:54.832346 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:54.836146 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.837309 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.837320 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.837322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.837600 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.838254 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.838261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.838264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.838899 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:54.839229 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:54.839620 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 15:50:54.839938 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":503,"threads_started":1}
I20260928 15:50:54.842012 27129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:54.930132 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 27005
I20260928 15:50:54.947582 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 27140
Sep 28 15:50:54 dist-test-slave-1vw0 krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:50:54.969856 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:54.948954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.22.157.195:35667)
0928 15:50:54.949777 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:54.949785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:54.949819 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:50:54.949884 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:54.949891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:54.950137 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:54.950307 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:54.952083 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.953641 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.955367 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:54.956194 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:54.956338 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:54.961727 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:54.961746 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:54.961768 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:54.961821 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:54.965687 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.966873 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.966886 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.966890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.967085 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:54.967658 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:54.967663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:54.967666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:54.968175 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:54.968527 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:54.969201 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 15:50:54.969527 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":374,"threads_started":1}
I20260928 15:50:54.971365 27264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:55.061411 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 27140
I20260928 15:50:55.078787 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 26775
I20260928 15:50:55.080292 26838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:55.192222 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 26775
2026-09-28T15:50:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4209 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:50:55 dist-test-slave-1vw0 krb5kdc[27289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:50:55 dist-test-slave-1vw0 krb5kdc[27289](info): set up 2 sockets
Sep 28 15:50:55 dist-test-slave-1vw0 krb5kdc[27289](info): commencing operation
krb5kdc: starting...
W20260928 15:50:56.342218 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:50:56 dist-test-slave-1vw0 krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:50:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:50:56.577328 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45893
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:45893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:56.922021 27302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:56.922456 27302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:56.922644 27302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:56.930258 27302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:56.930452 27302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:56.930541 27302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:56.930629 27302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:56.930703 27302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:56.941099 27302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:45893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:56.942646 27302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:56.944169 27302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:56.956921 27308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:56.956984 27307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:56.956897 27310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:56.957950 27302 server_base.cc:1034] running on GCE node
I20260928 15:50:56.958889 27302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:56.960415 27302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:56.961678 27302 hybrid_clock.cc:648] HybridClock initialized: now 1790610656961621 us; error 46 us; skew 500 ppm
I20260928 15:50:56.962087 27302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:56 dist-test-slave-1vw0 krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610656, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:56.965633 27302 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:50:56.966961 27302 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:56.967139 27302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:56.967499 27302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:56.970738 27302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "9d11a43e435447b7a64e892438642710"
format_stamp: "Formatted at 2026-09-28 15:50:56 on dist-test-slave-1vw0"
server_key: "aaf2bc7a7f49953abe5fbcf9ccb81b47"
server_key_iv: "54667a7f0d38bd663f06d4ac4a2d888b"
server_key_version: "encryptionkey@0"
I20260928 15:50:56.971607 27302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "9d11a43e435447b7a64e892438642710"
format_stamp: "Formatted at 2026-09-28 15:50:56 on dist-test-slave-1vw0"
server_key: "aaf2bc7a7f49953abe5fbcf9ccb81b47"
server_key_iv: "54667a7f0d38bd663f06d4ac4a2d888b"
server_key_version: "encryptionkey@0"
I20260928 15:50:56.977378 27302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:50:56.981206 27316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:56.982750 27302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:50:56.982983 27302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "9d11a43e435447b7a64e892438642710"
format_stamp: "Formatted at 2026-09-28 15:50:56 on dist-test-slave-1vw0"
server_key: "aaf2bc7a7f49953abe5fbcf9ccb81b47"
server_key_iv: "54667a7f0d38bd663f06d4ac4a2d888b"
server_key_version: "encryptionkey@0"
I20260928 15:50:56.983246 27302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:56.997928 27302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:57.003218 27302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:57.003573 27302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:57.024268 27302 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:45893
I20260928 15:50:57.024302 27368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:45893 every 8 connection(s)
I20260928 15:50:57.025326 27302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:50:57.029426 27369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:57.030506 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 27302
I20260928 15:50:57.030754 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:50:57.031450 23159 external_mini_cluster.cc:1428] Setting key 80d896505563bf10947596d3e692316d
I20260928 15:50:57.042649 27369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710: Bootstrap starting.
Sep 28 15:50:57 dist-test-slave-1vw0 krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:57.046330 27369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:57.047456 27369 log.cc:824] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:57.051172 27369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710: No bootstrap required, opened a new log
I20260928 15:50:57.055676 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:57.033673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.22.157.254:45893)
0928 15:50:57.034547 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:57.034559 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:57.034611 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:50:57.036204 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:57.036228 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:57.036666 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:57.037172 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:57.038694 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:57.040128 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:57.041606 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:57.042280 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:57.042511 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:57.047102 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:57.047111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:57.047128 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:57.047169 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:57.050910 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:57.051839 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:57.051846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:57.051853 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:57.052033 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:57.052483 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:57.052493 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:57.052495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:57.053191 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:57.053556 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:57.054077 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 15:50:57.054578 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":402,"threads_started":1}
I20260928 15:50:57.060324 27369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d11a43e435447b7a64e892438642710" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45893 } }
I20260928 15:50:57.060786 27369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:57.060911 27369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d11a43e435447b7a64e892438642710, State: Initialized, Role: FOLLOWER
I20260928 15:50:57.061568 27369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d11a43e435447b7a64e892438642710" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45893 } }
I20260928 15:50:57.061823 27369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:50:57.061981 27369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:50:57.062160 27369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:57.064716 27369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d11a43e435447b7a64e892438642710" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45893 } }
I20260928 15:50:57.065346 27369 leader_election.cc:302] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d11a43e435447b7a64e892438642710; no voters:
I20260928 15:50:57.065968 27369 leader_election.cc:288] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:50:57.066131 27374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:57.066867 27374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [term 1 LEADER]: Becoming Leader. State: Replica: 9d11a43e435447b7a64e892438642710, State: Running, Role: LEADER
I20260928 15:50:57.067451 27374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d11a43e435447b7a64e892438642710" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45893 } }
I20260928 15:50:57.068259 27369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:50:57.070371 27375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d11a43e435447b7a64e892438642710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d11a43e435447b7a64e892438642710" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45893 } } }
I20260928 15:50:57.071147 27375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:57.072067 27380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:50:57.071817 27376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d11a43e435447b7a64e892438642710. Latest consensus state: current_term: 1 leader_uuid: "9d11a43e435447b7a64e892438642710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d11a43e435447b7a64e892438642710" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45893 } } }
I20260928 15:50:57.073189 27376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710 [sys.catalog]: This master's current role is: LEADER
I20260928 15:50:57.077770 27380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:50:57.089607 27380 catalog_manager.cc:1347] Generated new cluster ID: eb90793ce527479bbe9f82b3020bc631
I20260928 15:50:57.089754 27380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:50:57.105634 27380 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:50:57.107019 27380 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:50:57.116902 27380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710: Generated new TSK 0
I20260928 15:50:57.117894 27380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:50:57.182518 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45893
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:57.535699 27395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:57.536012 27395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:57.536114 27395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:57.543828 27395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:57.543954 27395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:57.544075 27395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:50:57.554677 27395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:57.556164 27395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:57.557663 27395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:57.571839 27400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:57.571754 27401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:57.571810 27403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:57.572469 27395 server_base.cc:1034] running on GCE node
I20260928 15:50:57.573243 27395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:57.574342 27395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:57.575639 27395 hybrid_clock.cc:648] HybridClock initialized: now 1790610657575591 us; error 45 us; skew 500 ppm
I20260928 15:50:57.576030 27395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:57 dist-test-slave-1vw0 krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610657, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:57.579741 27395 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:50:57.581084 27395 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:57.581183 27395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:57.581482 27395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:57.584506 27395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "5fdcff28645b439d9d687b273fe0f61e"
format_stamp: "Formatted at 2026-09-28 15:50:57 on dist-test-slave-1vw0"
server_key: "fa84adf956a3504ef100366f2ac4cce6"
server_key_iv: "9bc0bd7d605bceaa2e0938df274bce17"
server_key_version: "encryptionkey@0"
I20260928 15:50:57.585425 27395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "5fdcff28645b439d9d687b273fe0f61e"
format_stamp: "Formatted at 2026-09-28 15:50:57 on dist-test-slave-1vw0"
server_key: "fa84adf956a3504ef100366f2ac4cce6"
server_key_iv: "9bc0bd7d605bceaa2e0938df274bce17"
server_key_version: "encryptionkey@0"
I20260928 15:50:57.591090 27395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 15:50:57.595022 27409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:57.596704 27395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:50:57.596997 27395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "5fdcff28645b439d9d687b273fe0f61e"
format_stamp: "Formatted at 2026-09-28 15:50:57 on dist-test-slave-1vw0"
server_key: "fa84adf956a3504ef100366f2ac4cce6"
server_key_iv: "9bc0bd7d605bceaa2e0938df274bce17"
server_key_version: "encryptionkey@0"
I20260928 15:50:57.597261 27395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:57.622175 27395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:57.627393 27395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:57.627722 27395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:57.628918 27395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:57.630460 27395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:57.630559 27395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:57.630676 27395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:57.630751 27395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:57.666996 27395 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:41629
I20260928 15:50:57.667128 27521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:41629 every 8 connection(s)
I20260928 15:50:57.667897 27395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:50:57.676584 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 27395
I20260928 15:50:57.676851 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:50:57.677711 23159 external_mini_cluster.cc:1428] Setting key d0ae87d37c897a64db2a1c4500eee6cc
Sep 28 15:50:57 dist-test-slave-1vw0 krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610657, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:57.692625 27524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:57.670787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:57533 (local address 127.22.157.254:45893)
0928 15:50:57.671264 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:57.671270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:57.672301 (+ 1031us) server_negotiation.cc:407] Connection header received
0928 15:50:57.674282 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:57.674287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:57.674392 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:57.674515 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:57.676183 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:57.677183 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:57.679501 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:57.680189 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:57.680306 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:57.684966 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:57.684977 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:57.684985 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:57.685038 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:57.688659 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:57.689611 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:57.689619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:57.689626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:57.689785 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:57.690261 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:57.690269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:57.690275 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:57.690683 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:57.690916 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:57.691835 (+ 919us) server_negotiation.cc:299] Negotiation successful
0928 15:50:57.692131 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260928 15:50:57.693473 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:57.671416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45893 (local address 127.22.157.193:57533)
0928 15:50:57.672129 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:57.672173 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:57.673951 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:57.674780 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:57.674789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:57.675348 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:57.675971 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:57.675990 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:57.677359 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:57.677365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:57.679289 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:57.679306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:57.680461 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:57.680469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:57.680838 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:57.681689 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:57.681728 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:57.684620 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:57.688898 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:57.688932 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:57.688948 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:57.689462 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:57.689929 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:57.689933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:57.689935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:57.690133 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:57.690887 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:57.690894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:57.691387 (+ 493us) client_negotiation.cc:769] Sending connection context
0928 15:50:57.691684 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 15:50:57.692161 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":297,"threads_started":1}
I20260928 15:50:57.695384 27522 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45893
I20260928 15:50:57.695755 27522 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:57.696527 27522 heartbeater.cc:511] Master 127.22.157.254:45893 requested a full tablet report, sending...
I20260928 15:50:57.699201 27333 ts_manager.cc:194] Registered new tserver with Master: 5fdcff28645b439d9d687b273fe0f61e (127.22.157.193:41629)
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
I20260928 15:50:57.700932 27333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:57533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:50:57.732869 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45893
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:58.080917 27528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:58.081243 27528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:58.081346 27528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:58.089140 27528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:58.089270 27528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:58.089393 27528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:50:58.099901 27528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:58.101444 27528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:58.102872 27528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:58.117226 27533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:58.117259 27534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:58.117311 27536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:58.118064 27528 server_base.cc:1034] running on GCE node
I20260928 15:50:58.118666 27528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:58.119647 27528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:58.120844 27528 hybrid_clock.cc:648] HybridClock initialized: now 1790610658120780 us; error 49 us; skew 500 ppm
I20260928 15:50:58.121161 27528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:58 dist-test-slave-1vw0 krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610658, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:58.124604 27528 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:50:58.125929 27528 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:58.126106 27528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:58.126448 27528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:58.129659 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "97663332d0ea476c8bb6badb9ec64568"
format_stamp: "Formatted at 2026-09-28 15:50:58 on dist-test-slave-1vw0"
server_key: "a676aaa51ea2f5c1ef31c89f60e00b28"
server_key_iv: "a3818da9b4488d2db563a40718119f2f"
server_key_version: "encryptionkey@0"
I20260928 15:50:58.130550 27528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "97663332d0ea476c8bb6badb9ec64568"
format_stamp: "Formatted at 2026-09-28 15:50:58 on dist-test-slave-1vw0"
server_key: "a676aaa51ea2f5c1ef31c89f60e00b28"
server_key_iv: "a3818da9b4488d2db563a40718119f2f"
server_key_version: "encryptionkey@0"
I20260928 15:50:58.136072 27528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:50:58.139878 27542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:58.141396 27528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:50:58.141584 27528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "97663332d0ea476c8bb6badb9ec64568"
format_stamp: "Formatted at 2026-09-28 15:50:58 on dist-test-slave-1vw0"
server_key: "a676aaa51ea2f5c1ef31c89f60e00b28"
server_key_iv: "a3818da9b4488d2db563a40718119f2f"
server_key_version: "encryptionkey@0"
I20260928 15:50:58.141826 27528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:58.156564 27528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:58.161187 27528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:58.161484 27528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:58.162504 27528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:58.163966 27528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:58.164094 27528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:58.164232 27528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:58.164325 27528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:58.200882 27528 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:36363
I20260928 15:50:58.200925 27657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:36363 every 8 connection(s)
I20260928 15:50:58.201810 27528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:50:58.205838 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 27528
I20260928 15:50:58.206074 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:50:58.206665 23159 external_mini_cluster.cc:1428] Setting key 8c5c808f3488dfebc51be2b54aca2102
Sep 28 15:50:58 dist-test-slave-1vw0 krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610658, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:58.223748 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.204810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:51605 (local address 127.22.157.254:45893)
0928 15:50:58.205271 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:58.205275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:58.206399 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 15:50:58.208542 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:58.208547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:58.208633 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:58.208793 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:58.210380 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.211136 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.212779 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.213225 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.213377 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.217530 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:58.217536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:58.217539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:58.217576 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:58.220317 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:58.221240 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:58.221246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:58.221254 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:58.221385 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:58.221864 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:58.221873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:58.221876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:58.222161 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:58.222327 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:58.223122 (+ 795us) server_negotiation.cc:299] Negotiation successful
0928 15:50:58.223360 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":243,"threads_started":1}
I20260928 15:50:58.224933 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.205354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45893 (local address 127.22.157.194:51605)
0928 15:50:58.206211 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:58.206252 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:58.208254 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:58.209056 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:58.209065 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:58.209601 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:58.210184 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.210201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.211296 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.211302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.212597 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.212610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.213475 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.213482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.213721 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.214482 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:58.214510 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:58.217241 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:58.220505 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:58.220524 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:58.220550 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:58.221096 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:58.221510 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:58.221523 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:58.221526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:58.221729 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:58.222300 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:58.222310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:58.222709 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 15:50:58.223000 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 15:50:58.223601 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":354,"threads_started":1}
I20260928 15:50:58.226576 27658 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45893
I20260928 15:50:58.226966 27658 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
I20260928 15:50:58.227993 27658 heartbeater.cc:511] Master 127.22.157.254:45893 requested a full tablet report, sending...
I20260928 15:50:58.230259 27333 ts_manager.cc:194] Registered new tserver with Master: 97663332d0ea476c8bb6badb9ec64568 (127.22.157.194:36363)
I20260928 15:50:58.231165 27333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:51605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:50:58.261804 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45893
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:50:58.603163 27664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:58.603482 27664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:58.603673 27664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:58.611528 27664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:58.611673 27664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:58.611886 27664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:50:58.622721 27664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:58.624191 27664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:58.625715 27664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:50:58.641423 27672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:58.641467 27670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:50:58.641551 27669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:50:58.643102 27664 server_base.cc:1034] running on GCE node
I20260928 15:50:58.643939 27664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:50:58.645160 27664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:50:58.646453 27664 hybrid_clock.cc:648] HybridClock initialized: now 1790610658646374 us; error 67 us; skew 500 ppm
I20260928 15:50:58.646895 27664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:50:58 dist-test-slave-1vw0 krb5kdc[27289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610658, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:50:58.650986 27664 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:50:58.652431 27664 fs_manager.cc:359] Metadata directory not provided
I20260928 15:50:58.652554 27664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:50:58.653076 27664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:50:58.656533 27664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "08b7fc3dd2fd425ab6d2860fe947d85c"
format_stamp: "Formatted at 2026-09-28 15:50:58 on dist-test-slave-1vw0"
server_key: "5d39f92202f16271480e720c8b90c186"
server_key_iv: "7d37fbb454246f7d9ef1396895ce4bc3"
server_key_version: "encryptionkey@0"
I20260928 15:50:58.657559 27664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "08b7fc3dd2fd425ab6d2860fe947d85c"
format_stamp: "Formatted at 2026-09-28 15:50:58 on dist-test-slave-1vw0"
server_key: "5d39f92202f16271480e720c8b90c186"
server_key_iv: "7d37fbb454246f7d9ef1396895ce4bc3"
server_key_version: "encryptionkey@0"
I20260928 15:50:58.663846 27664 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:50:58.668633 27678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:58.670980 27664 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:50:58.671213 27664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "08b7fc3dd2fd425ab6d2860fe947d85c"
format_stamp: "Formatted at 2026-09-28 15:50:58 on dist-test-slave-1vw0"
server_key: "5d39f92202f16271480e720c8b90c186"
server_key_iv: "7d37fbb454246f7d9ef1396895ce4bc3"
server_key_version: "encryptionkey@0"
I20260928 15:50:58.671521 27664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:50:58.699560 27664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:50:58.704511 27522 heartbeater.cc:503] Master 127.22.157.254:45893 was elected leader, sending a full tablet report...
I20260928 15:50:58.704952 27664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:50:58.705268 27664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:50:58.706568 27664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:50:58.708161 27664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:50:58.708297 27664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:58.708446 27664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:50:58.708540 27664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:50:58.748037 27664 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:44649
I20260928 15:50:58.748080 27791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:44649 every 8 connection(s)
I20260928 15:50:58.749089 27664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:50:58.755555 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 27664
I20260928 15:50:58.755817 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:50:58.756381 23159 external_mini_cluster.cc:1428] Setting key 7713d30828db485b62245826a1baebac
Sep 28 15:50:58 dist-test-slave-1vw0 krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610658, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:50:58.773960 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.751904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:50657 (local address 127.22.157.254:45893)
0928 15:50:58.752506 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:58.752513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:58.753795 (+ 1282us) server_negotiation.cc:407] Connection header received
0928 15:50:58.756096 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:58.756103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:58.756218 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:58.756354 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:58.758669 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.759744 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.761721 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.762187 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.762295 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.766502 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:58.766508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:58.766511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:58.766549 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:58.769588 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:58.770606 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:58.770610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:58.770613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:58.770751 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:58.771254 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:58.771258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:58.771261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:58.771686 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:58.771898 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:58.773257 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0928 15:50:58.773564 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":253,"threads_started":1}
I20260928 15:50:58.775151 27795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.752647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45893 (local address 127.22.157.195:50657)
0928 15:50:58.753561 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:58.753642 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:58.755705 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:58.756772 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:58.756790 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:58.757615 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:50:58.758400 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.758425 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.759944 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.759948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.761553 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.761567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.762410 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.762415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.762682 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.763440 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:50:58.763471 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:50:58.766232 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:50:58.769820 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:58.769831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:58.769852 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:58.770408 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:58.770900 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:50:58.770904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:50:58.770907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:50:58.771105 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:50:58.771962 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:50:58.771973 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:50:58.772700 (+ 727us) client_negotiation.cc:769] Sending connection context
0928 15:50:58.773160 (+ 460us) client_negotiation.cc:240] Negotiation successful
0928 15:50:58.773783 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":380,"threads_started":1}
I20260928 15:50:58.777940 27792 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45893
I20260928 15:50:58.778482 27792 heartbeater.cc:464] Registering TS with master...
I20260928 15:50:58.779320 27792 heartbeater.cc:511] Master 127.22.157.254:45893 requested a full tablet report, sending...
I20260928 15:50:58.781263 27333 ts_manager.cc:194] Registered new tserver with Master: 08b7fc3dd2fd425ab6d2860fe947d85c (127.22.157.195:44649)
I20260928 15:50:58.782240 27333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:50657
I20260928 15:50:58.786885 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:50:58.807349 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.791999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.22.157.254:45893)
0928 15:50:58.792198 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:58.792203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:58.792925 (+ 722us) server_negotiation.cc:407] Connection header received
0928 15:50:58.793111 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:58.793115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:58.793193 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:58.793292 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:58.795150 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.795954 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.797552 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.797990 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.798112 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.801040 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:58.801045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:58.801049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:58.801094 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:58.804128 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:58.805059 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:58.805063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:58.805065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:58.805210 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:58.805775 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:58.805780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:58.805783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:58.806173 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:58.806455 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:58.806863 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 15:50:58.807092 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:50:58.812944 27333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:50:58.820324 27333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:50:58.859041 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.834599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.22.157.193:41629)
0928 15:50:58.848012 (+ 13413us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:58.848027 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:58.849233 (+ 1206us) server_negotiation.cc:407] Connection header received
0928 15:50:58.849398 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:58.849413 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:58.849744 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:58.850043 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:58.851579 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.853312 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.856066 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.857667 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.857895 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.858108 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:58.858483 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 15:50:58.858659 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13128,"thread_start_us":13427,"threads_started":1}
I20260928 15:50:58.859071 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.833954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:41629 (local address 127.0.0.1:42300)
0928 15:50:58.848814 (+ 14860us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:58.848838 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:58.849039 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:58.850824 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:58.850835 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:58.850888 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:58.851312 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.851328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.853529 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.853535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.855850 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.855866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.857923 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.857929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.858168 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.858236 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 15:50:58.858396 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 15:50:58.858626 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14698,"mutex_wait_us":4,"thread_start_us":6488,"threads_started":1}
I20260928 15:50:58.860858 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.841887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.22.157.194:36363)
0928 15:50:58.850680 (+ 8793us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:58.850688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:58.851936 (+ 1248us) server_negotiation.cc:407] Connection header received
0928 15:50:58.852091 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:58.852097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:58.852289 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:58.852530 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:58.853548 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.854681 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.857976 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.859601 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.859790 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.859978 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:58.860274 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 15:50:58.860470 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8639,"thread_start_us":8556,"threads_started":1}
I20260928 15:50:58.861451 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.846370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:36363 (local address 127.0.0.1:52718)
0928 15:50:58.851723 (+ 5353us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:58.851748 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:58.851922 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:58.852824 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:58.852829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:58.852857 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:58.853324 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.853336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.854864 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.854872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.857757 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.857766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.859850 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.859857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.860032 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.860063 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 15:50:58.860152 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 15:50:58.860301 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5217,"mutex_wait_us":228,"thread_start_us":2099,"threads_started":1}
I20260928 15:50:58.861775 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.833857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:44649 (local address 127.0.0.1:48704)
0928 15:50:58.840921 (+ 7064us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:58.840971 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:58.841201 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:58.852015 (+ 10814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:58.852021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:58.852049 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:58.852425 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.852438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.854988 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.854993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.857724 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.857748 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.860460 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.860468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.860686 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.860751 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 15:50:58.860829 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:50:58.861200 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6762,"mutex_wait_us":170,"thread_start_us":17558,"threads_started":1}
I20260928 15:50:58.862785 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.834326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.22.157.195:44649)
0928 15:50:58.851261 (+ 16935us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:58.851271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:58.851308 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:50:58.851410 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:58.851418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:58.851727 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:58.852629 (+ 902us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:58.852969 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.854761 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.857965 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.860206 (+ 2241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.862023 (+ 1817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.862201 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:58.862272 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 15:50:58.862455 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16736,"thread_start_us":16562,"threads_started":1}
I20260928 15:50:58.863857 27457 tablet_service.cc:1467] Processing CreateTablet for tablet 5441ab1abb22419b9c33240c6685c8de (DEFAULT_TABLE table=test-table [id=341c4f7d701c45598c64c2cf3d6bacab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:58.865300 27457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5441ab1abb22419b9c33240c6685c8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:58.865383 27592 tablet_service.cc:1467] Processing CreateTablet for tablet 5441ab1abb22419b9c33240c6685c8de (DEFAULT_TABLE table=test-table [id=341c4f7d701c45598c64c2cf3d6bacab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:58.866888 27592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5441ab1abb22419b9c33240c6685c8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:58.866549 27727 tablet_service.cc:1467] Processing CreateTablet for tablet 5441ab1abb22419b9c33240c6685c8de (DEFAULT_TABLE table=test-table [id=341c4f7d701c45598c64c2cf3d6bacab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:50:58.868845 27727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5441ab1abb22419b9c33240c6685c8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:50:58.879153 27811 tablet_bootstrap.cc:492] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e: Bootstrap starting.
I20260928 15:50:58.883231 27811 tablet_bootstrap.cc:654] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:58.883599 27812 tablet_bootstrap.cc:492] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: Bootstrap starting.
I20260928 15:50:58.884928 27811 log.cc:824] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:58.887645 27812 tablet_bootstrap.cc:654] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:58.888787 27811 tablet_bootstrap.cc:492] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e: No bootstrap required, opened a new log
I20260928 15:50:58.889065 27812 log.cc:824] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:58.889150 27811 ts_tablet_manager.cc:1397] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260928 15:50:58.886538 27813 tablet_bootstrap.cc:492] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: Bootstrap starting.
I20260928 15:50:58.893808 27813 tablet_bootstrap.cc:654] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: Neither blocks nor log segments found. Creating new log.
I20260928 15:50:58.894986 27812 tablet_bootstrap.cc:492] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: No bootstrap required, opened a new log
I20260928 15:50:58.895222 27813 log.cc:824] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:50:58.895314 27812 ts_tablet_manager.cc:1397] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260928 15:50:58.898875 27813 tablet_bootstrap.cc:492] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: No bootstrap required, opened a new log
I20260928 15:50:58.899178 27813 ts_tablet_manager.cc:1397] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260928 15:50:58.899865 27811 raft_consensus.cc:348] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } }
I20260928 15:50:58.900369 27811 raft_consensus.cc:374] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:58.900480 27811 raft_consensus.cc:729] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fdcff28645b439d9d687b273fe0f61e, State: Initialized, Role: FOLLOWER
I20260928 15:50:58.901202 27811 consensus_queue.cc:260] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } }
I20260928 15:50:58.903888 27812 raft_consensus.cc:348] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } }
I20260928 15:50:58.904299 27812 raft_consensus.cc:374] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:58.904414 27812 raft_consensus.cc:729] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97663332d0ea476c8bb6badb9ec64568, State: Initialized, Role: FOLLOWER
I20260928 15:50:58.905121 27812 consensus_queue.cc:260] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } }
I20260928 15:50:58.905820 27811 ts_tablet_manager.cc:1428] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260928 15:50:58.906252 27658 heartbeater.cc:503] Master 127.22.157.254:45893 was elected leader, sending a full tablet report...
I20260928 15:50:58.908213 27812 ts_tablet_manager.cc:1428] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
I20260928 15:50:58.912356 27813 raft_consensus.cc:348] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } }
I20260928 15:50:58.912802 27813 raft_consensus.cc:374] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:50:58.912930 27813 raft_consensus.cc:729] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b7fc3dd2fd425ab6d2860fe947d85c, State: Initialized, Role: FOLLOWER
I20260928 15:50:58.913529 27813 consensus_queue.cc:260] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } }
I20260928 15:50:58.914597 27792 heartbeater.cc:503] Master 127.22.157.254:45893 was elected leader, sending a full tablet report...
I20260928 15:50:58.915215 27813 ts_tablet_manager.cc:1428] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260928 15:50:58.916550 27817 raft_consensus.cc:482] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:50:58.916850 27817 raft_consensus.cc:504] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } }
I20260928 15:50:58.918838 27817 leader_election.cc:288] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08b7fc3dd2fd425ab6d2860fe947d85c (127.22.157.195:44649), 97663332d0ea476c8bb6badb9ec64568 (127.22.157.194:36363)
W20260928 15:50:58.920598 27523 tablet.cc:2369] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:58.926254 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.919557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:49461 (local address 127.22.157.195:44649)
0928 15:50:58.919807 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:58.919816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:58.920273 (+ 457us) server_negotiation.cc:407] Connection header received
0928 15:50:58.920530 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:58.920535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:58.920637 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:58.920707 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:58.921694 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.922444 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.924360 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.925257 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.925456 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.925534 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:58.925834 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 15:50:58.925966 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:50:58.926400 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.919034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:44649 (local address 127.22.157.193:49461)
0928 15:50:58.920159 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:58.920178 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:58.920353 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:58.920961 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:58.920965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:58.920998 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:58.921501 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.921516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.922598 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.922604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.924192 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.924204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.925472 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.925478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.925667 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.925691 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:50:58.925809 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:50:58.925965 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":1,"thread_start_us":1532,"threads_started":1}
I20260928 15:50:58.926644 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.919887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:47303 (local address 127.22.157.194:36363)
0928 15:50:58.920116 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:58.920122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:58.921426 (+ 1304us) server_negotiation.cc:407] Connection header received
0928 15:50:58.921651 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:58.921656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:58.921725 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:58.921834 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:50:58.922350 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.923074 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.924884 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.925665 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.925851 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.925931 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:58.926249 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 15:50:58.926382 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:50:58.926879 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.919114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:36363 (local address 127.22.157.193:47303)
0928 15:50:58.921307 (+ 2193us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:58.921323 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:50:58.921504 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:50:58.921847 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:50:58.921852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:50:58.921876 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:50:58.922190 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.922198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.923223 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.923228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.924671 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:50:58.924684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.925905 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.925912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:50:58.926073 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.926100 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 15:50:58.926210 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:50:58.926352 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2079,"mutex_wait_us":20,"thread_start_us":1873,"threads_started":1}
I20260928 15:50:58.927465 27747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5441ab1abb22419b9c33240c6685c8de" candidate_uuid: "5fdcff28645b439d9d687b273fe0f61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" is_pre_election: true
I20260928 15:50:58.927927 27612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5441ab1abb22419b9c33240c6685c8de" candidate_uuid: "5fdcff28645b439d9d687b273fe0f61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97663332d0ea476c8bb6badb9ec64568" is_pre_election: true
I20260928 15:50:58.928107 27747 raft_consensus.cc:2454] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fdcff28645b439d9d687b273fe0f61e in term 0.
I20260928 15:50:58.928460 27612 raft_consensus.cc:2454] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fdcff28645b439d9d687b273fe0f61e in term 0.
I20260928 15:50:58.928996 27412 leader_election.cc:302] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b7fc3dd2fd425ab6d2860fe947d85c, 5fdcff28645b439d9d687b273fe0f61e; no voters:
I20260928 15:50:58.929498 27817 raft_consensus.cc:2781] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:50:58.929677 27817 raft_consensus.cc:482] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:50:58.929759 27817 raft_consensus.cc:3037] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:58.932107 27817 raft_consensus.cc:504] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } }
I20260928 15:50:58.932932 27817 leader_election.cc:288] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [CANDIDATE]: Term 1 election: Requested vote from peers 08b7fc3dd2fd425ab6d2860fe947d85c (127.22.157.195:44649), 97663332d0ea476c8bb6badb9ec64568 (127.22.157.194:36363)
I20260928 15:50:58.933365 27747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5441ab1abb22419b9c33240c6685c8de" candidate_uuid: "5fdcff28645b439d9d687b273fe0f61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c"
I20260928 15:50:58.933439 27612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5441ab1abb22419b9c33240c6685c8de" candidate_uuid: "5fdcff28645b439d9d687b273fe0f61e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97663332d0ea476c8bb6badb9ec64568"
I20260928 15:50:58.933606 27747 raft_consensus.cc:3037] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:58.933657 27612 raft_consensus.cc:3037] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:50:58.935968 27612 raft_consensus.cc:2454] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fdcff28645b439d9d687b273fe0f61e in term 1.
I20260928 15:50:58.936017 27747 raft_consensus.cc:2454] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fdcff28645b439d9d687b273fe0f61e in term 1.
I20260928 15:50:58.936561 27413 leader_election.cc:302] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fdcff28645b439d9d687b273fe0f61e, 97663332d0ea476c8bb6badb9ec64568; no voters:
I20260928 15:50:58.937048 27817 raft_consensus.cc:2781] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:50:58.937619 27817 raft_consensus.cc:686] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 1 LEADER]: Becoming Leader. State: Replica: 5fdcff28645b439d9d687b273fe0f61e, State: Running, Role: LEADER
I20260928 15:50:58.938159 27817 consensus_queue.cc:237] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } }
I20260928 15:50:58.943631 27333 catalog_manager.cc:5495] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fdcff28645b439d9d687b273fe0f61e (127.22.157.193). New cstate: current_term: 1 leader_uuid: "5fdcff28645b439d9d687b273fe0f61e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fdcff28645b439d9d687b273fe0f61e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 41629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 } health_report { overall_health: UNKNOWN } } }
W20260928 15:50:58.954054 27659 tablet.cc:2369] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:50:58.965054 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:58.959444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.22.157.193:41629)
0928 15:50:58.959678 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:58.959683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:58.959904 (+ 221us) server_negotiation.cc:407] Connection header received
0928 15:50:58.960113 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:58.960118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:58.960196 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:58.960325 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:50:58.960700 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.961492 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.962856 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:58.963266 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:58.963361 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:58.963863 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:50:58.964378 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:50:58.964498 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:58.964613 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 15:50:58.964681 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:50:58.973744 27747 raft_consensus.cc:1261] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [term 1 FOLLOWER]: Refusing update from remote peer 5fdcff28645b439d9d687b273fe0f61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:58.973744 27612 raft_consensus.cc:1261] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [term 1 FOLLOWER]: Refusing update from remote peer 5fdcff28645b439d9d687b273fe0f61e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:50:58.974848 27822 consensus_queue.cc:1035] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b7fc3dd2fd425ab6d2860fe947d85c" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:58.975234 27817 consensus_queue.cc:1035] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [LEADER]: Connected to new peer: Peer: permanent_uuid: "97663332d0ea476c8bb6badb9ec64568" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:50:58.985440 27825 mvcc.cc:205] Tried to move back new op lower bound from 7334341259145973760 to 7334341259014901760. Current Snapshot: MvccSnapshot[applied={T|T < 7334341259145973760}]
I20260928 15:50:58.987246 27827 mvcc.cc:205] Tried to move back new op lower bound from 7334341259145973760 to 7334341259014901760. Current Snapshot: MvccSnapshot[applied={T|T < 7334341259145973760}]
I20260928 15:50:59.004244 27332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:59.004714 27332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:50:59.008899 27332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d11a43e435447b7a64e892438642710: Sending DeleteTablet for 3 replicas of tablet 5441ab1abb22419b9c33240c6685c8de
I20260928 15:50:59.010389 27592 tablet_service.cc:1514] Processing DeleteTablet for tablet 5441ab1abb22419b9c33240c6685c8de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:59 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:52718
I20260928 15:50:59.010439 27727 tablet_service.cc:1514] Processing DeleteTablet for tablet 5441ab1abb22419b9c33240c6685c8de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:59 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:48704
I20260928 15:50:59.011260 27457 tablet_service.cc:1514] Processing DeleteTablet for tablet 5441ab1abb22419b9c33240c6685c8de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:50:59 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:42300
I20260928 15:50:59.012058 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 27395
I20260928 15:50:59.022903 27834 tablet_replica.cc:330] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: stopping tablet replica
I20260928 15:50:59.023552 27834 raft_consensus.cc:2229] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:59.024499 27835 tablet_replica.cc:330] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: stopping tablet replica
I20260928 15:50:59.024952 27835 raft_consensus.cc:2229] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:50:59.025487 27835 raft_consensus.cc:2258] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:59.026120 27834 raft_consensus.cc:2258] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:59.028362 27835 ts_tablet_manager.cc:1905] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:59.031796 27834 ts_tablet_manager.cc:1905] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:59.037012 27835 ts_tablet_manager.cc:1918] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:50:59.037456 27836 tablet_replica.cc:330] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e: stopping tablet replica
Sep 28 15:50:59 dist-test-slave-1vw0 krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:50:59.040326 27835 log.cc:1196] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/5441ab1abb22419b9c33240c6685c8de
I20260928 15:50:59.040815 27836 raft_consensus.cc:2229] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:50:59.041978 27836 raft_consensus.cc:2258] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:50:59.043274 27834 ts_tablet_manager.cc:1918] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:50:59.043464 27834 log.cc:1196] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/5441ab1abb22419b9c33240c6685c8de
I20260928 15:50:59.044165 27834 ts_tablet_manager.cc:1939] T 5441ab1abb22419b9c33240c6685c8de P 97663332d0ea476c8bb6badb9ec64568: Deleting consensus metadata
I20260928 15:50:59.045897 27836 ts_tablet_manager.cc:1905] T 5441ab1abb22419b9c33240c6685c8de P 5fdcff28645b439d9d687b273fe0f61e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:50:59.046509 27835 ts_tablet_manager.cc:1939] T 5441ab1abb22419b9c33240c6685c8de P 08b7fc3dd2fd425ab6d2860fe947d85c: Deleting consensus metadata
I20260928 15:50:59.046464 27320 catalog_manager.cc:4841] TS 97663332d0ea476c8bb6badb9ec64568 (127.22.157.194:36363): tablet 5441ab1abb22419b9c33240c6685c8de (table test-table [id=341c4f7d701c45598c64c2cf3d6bacab]) successfully deleted
I20260928 15:50:59.049319 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.013711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.22.157.193:41629)
0928 15:50:59.013911 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:59.013918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:59.017233 (+ 3315us) server_negotiation.cc:407] Connection header received
0928 15:50:59.017568 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:59.017578 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:59.017687 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:59.017824 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:59.024375 (+ 6551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:59.025691 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:59.027699 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:59.028431 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:59.028556 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:59.039314 (+ 10758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:59.039331 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:59.039357 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:59.039429 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:59.043836 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:59.045421 (+ 1585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:59.045431 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:59.045435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:59.045695 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:59.046826 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:59.046832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:59.046835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:59.047355 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:59.047670 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:59.048360 (+ 690us) server_negotiation.cc:299] Negotiation successful
0928 15:50:59.048662 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:50:59.050081 27517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:59.050585 27319 catalog_manager.cc:4841] TS 08b7fc3dd2fd425ab6d2860fe947d85c (127.22.157.195:44649): tablet 5441ab1abb22419b9c33240c6685c8de (table test-table [id=341c4f7d701c45598c64c2cf3d6bacab]) successfully deleted
I20260928 15:50:59.151400 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 27395
W20260928 15:50:59.169759 27320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.157.193:41629 (error 108)
I20260928 15:50:59.170666 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 27528
I20260928 15:50:59.172161 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.171611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:41629 (local address 127.0.0.1:42336)
0928 15:50:59.171792 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:59.171954 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 15:50:59.172436 27320 catalog_manager.cc:4584] TS 5fdcff28645b439d9d687b273fe0f61e (127.22.157.193:41629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5441ab1abb22419b9c33240c6685c8de: Network error: Client connection negotiation failed: client connection to 127.22.157.193:41629: connect: Connection refused (error 111)
Sep 28 15:50:59 dist-test-slave-1vw0 krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:50:59.186712 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.171547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.22.157.194:36363)
0928 15:50:59.171755 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:59.171762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:59.171785 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:50:59.171969 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:59.171975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:59.172065 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:59.172185 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:59.173638 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:59.174419 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:59.175700 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:59.176198 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:59.176314 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:59.180950 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:59.180964 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:59.180978 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:59.181013 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:59.183754 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:59.184427 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:59.184434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:59.184437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:59.184606 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:59.185122 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:59.185126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:59.185128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:59.185501 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:59.185723 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:59.186112 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 15:50:59.186341 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:50:59.187534 27653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:59.226503 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.225875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:41629 (local address 127.0.0.1:42350)
0928 15:50:59.226155 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:59.226288 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 15:50:59.227763 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.227344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:41629 (local address 127.0.0.1:42354)
0928 15:50:59.227529 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:59.227631 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:50:59.288260 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 27528
I20260928 15:50:59.288801 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.288224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:41629 (local address 127.0.0.1:42370)
0928 15:50:59.288477 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:59.288584 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 15:50:59.290043 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.289563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:41629 (local address 127.0.0.1:42380)
0928 15:50:59.289769 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:59.289871 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 15:50:59.306979 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 27664
Sep 28 15:50:59 dist-test-slave-1vw0 krb5kdc[27289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:50:59.332792 27807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.315505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.22.157.195:44649)
0928 15:50:59.315791 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 15:50:59.315797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:50:59.315826 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:50:59.315926 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:50:59.315933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:50:59.316028 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:50:59.316188 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:50:59.317754 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:59.318505 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:59.319932 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:50:59.320344 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:50:59.320497 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:50:59.325064 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:50:59.325088 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:50:59.325110 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:50:59.325174 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:50:59.329283 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:59.330143 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:59.330155 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:59.330159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:59.330356 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:50:59.330792 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:50:59.330798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:50:59.330803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:50:59.331375 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:50:59.331695 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 15:50:59.332024 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 15:50:59.332303 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 15:50:59.333578 27787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:59.371146 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.370648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:41629 (local address 127.0.0.1:42384)
0928 15:50:59.370863 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:59.370980 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 15:50:59.372398 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.371935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:41629 (local address 127.0.0.1:42398)
0928 15:50:59.372173 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:59.372262 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 15:50:59.425915 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 27664
I20260928 15:50:59.442934 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 27302
I20260928 15:50:59.444298 27364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:50:59.555186 27806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:50:59.554642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:41629 (local address 127.0.0.1:42414)
0928 15:50:59.554869 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 15:50:59.554996 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:41629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 15:50:59.556075 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 27302
2026-09-28T15:50:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4358 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T15:50:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:50:59Z Disabled control of system clock
I20260928 15:50:59.628432 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:35615
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:35615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:50:59.973520 27846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:50:59.973845 27846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:50:59.973938 27846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:50:59.981995 27846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:50:59.982122 27846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:50:59.982160 27846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:50:59.982210 27846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:50:59.982251 27846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:50:59.991727 27846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:35615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:35615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:50:59.993229 27846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:50:59.995064 27846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:00.007887 27851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:00.007808 27852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:00.007861 27854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:00.008824 27846 server_base.cc:1034] running on GCE node
I20260928 15:51:00.009692 27846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:00.011337 27846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:00.012635 27846 hybrid_clock.cc:648] HybridClock initialized: now 1790610660012577 us; error 55 us; skew 500 ppm
I20260928 15:51:00.013036 27846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:00.015614 27846 webserver.cc:460] Webserver started at http://127.22.157.254:36415/ using document root <none> and password file <none>
I20260928 15:51:00.016376 27846 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:00.016530 27846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:00.016952 27846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:00.019981 27846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "c0d795832afd43c9a5e6c648247ca605"
format_stamp: "Formatted at 2026-09-28 15:51:00 on dist-test-slave-1vw0"
server_key: "7073ef648a2d6eee9cf7daeebd21decc"
server_key_iv: "7f9fde50cf37f18216bc69fbdf585853"
server_key_version: "encryptionkey@0"
I20260928 15:51:00.020925 27846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "c0d795832afd43c9a5e6c648247ca605"
format_stamp: "Formatted at 2026-09-28 15:51:00 on dist-test-slave-1vw0"
server_key: "7073ef648a2d6eee9cf7daeebd21decc"
server_key_iv: "7f9fde50cf37f18216bc69fbdf585853"
server_key_version: "encryptionkey@0"
I20260928 15:51:00.026578 27846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:51:00.030438 27861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:00.031935 27846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:51:00.032171 27846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "c0d795832afd43c9a5e6c648247ca605"
format_stamp: "Formatted at 2026-09-28 15:51:00 on dist-test-slave-1vw0"
server_key: "7073ef648a2d6eee9cf7daeebd21decc"
server_key_iv: "7f9fde50cf37f18216bc69fbdf585853"
server_key_version: "encryptionkey@0"
I20260928 15:51:00.032434 27846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:00.038800 27846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:00.039695 27846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:00.040042 27846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:00.059955 27846 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:35615
I20260928 15:51:00.059974 27912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:35615 every 8 connection(s)
I20260928 15:51:00.061528 27846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:00.065652 27913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:00.070921 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 27846
I20260928 15:51:00.071153 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:00.071835 23159 external_mini_cluster.cc:1428] Setting key 5a59c54ea00744c4b6ddf0c4970bf4e6
I20260928 15:51:00.077390 27913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605: Bootstrap starting.
I20260928 15:51:00.081257 27916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:00.074223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.22.157.254:35615)
0928 15:51:00.075417 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:00.075428 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:00.075471 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:51:00.077209 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:00.077233 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:00.077745 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:00.078127 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:00.079413 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:00.079420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:00.079433 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:00.079449 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:00.079529 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:00.079552 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:00.079734 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:00.079847 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 15:51:00.080185 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":445,"threads_started":1}
I20260928 15:51:00.081396 27913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:00.082513 27913 log.cc:824] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:00.085578 27913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605: No bootstrap required, opened a new log
I20260928 15:51:00.092813 27913 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d795832afd43c9a5e6c648247ca605" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35615 } }
I20260928 15:51:00.093276 27913 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:00.093392 27913 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d795832afd43c9a5e6c648247ca605, State: Initialized, Role: FOLLOWER
I20260928 15:51:00.094007 27913 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d795832afd43c9a5e6c648247ca605" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35615 } }
I20260928 15:51:00.094281 27913 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:00.094429 27913 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:00.094614 27913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:00.096896 27913 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d795832afd43c9a5e6c648247ca605" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35615 } }
I20260928 15:51:00.097443 27913 leader_election.cc:302] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d795832afd43c9a5e6c648247ca605; no voters:
I20260928 15:51:00.098168 27913 leader_election.cc:288] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:00.098353 27918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:00.099313 27918 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [term 1 LEADER]: Becoming Leader. State: Replica: c0d795832afd43c9a5e6c648247ca605, State: Running, Role: LEADER
I20260928 15:51:00.099879 27918 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d795832afd43c9a5e6c648247ca605" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35615 } }
I20260928 15:51:00.100490 27913 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:00.103232 27920 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d795832afd43c9a5e6c648247ca605. Latest consensus state: current_term: 1 leader_uuid: "c0d795832afd43c9a5e6c648247ca605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d795832afd43c9a5e6c648247ca605" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35615 } } }
I20260928 15:51:00.103274 27919 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0d795832afd43c9a5e6c648247ca605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d795832afd43c9a5e6c648247ca605" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35615 } } }
I20260928 15:51:00.103755 27919 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:00.103760 27920 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:00.108517 27925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:00.113576 27925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:00.122391 27925 catalog_manager.cc:1347] Generated new cluster ID: ae15b423713b484dbb88801d6302906a
I20260928 15:51:00.122541 27925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:00.140375 27925 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:00.140928 27925 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:00.152930 27925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605: Generated new TSK 0
I20260928 15:51:00.154177 27925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:51:00.172317 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35615
--builtin_ntp_servers=127.22.157.212:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 15:51:00.518498 27937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:00.518793 27937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:00.518888 27937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:00.526257 27937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:00.526367 27937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:00.526485 27937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:00.536029 27937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:00.537616 27937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:00.539419 27937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:00.554059 27942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:00.554040 27943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:00.554042 27945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:00.555226 27937 server_base.cc:1034] running on GCE node
I20260928 15:51:00.555909 27937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:00.556937 27937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:00.558184 27937 hybrid_clock.cc:648] HybridClock initialized: now 1790610660558126 us; error 49 us; skew 500 ppm
I20260928 15:51:00.558549 27937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:00.561019 27937 webserver.cc:460] Webserver started at http://127.22.157.193:33889/ using document root <none> and password file <none>
I20260928 15:51:00.561743 27937 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:00.561883 27937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:00.562237 27937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:00.565483 27937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "863ddae0b0a54a8fbe6582c2d9183473"
format_stamp: "Formatted at 2026-09-28 15:51:00 on dist-test-slave-1vw0"
server_key: "bb872310aa9be222843e900c3eb0c70f"
server_key_iv: "debe5c9835b5abd49deb97c78169d788"
server_key_version: "encryptionkey@0"
I20260928 15:51:00.566334 27937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "863ddae0b0a54a8fbe6582c2d9183473"
format_stamp: "Formatted at 2026-09-28 15:51:00 on dist-test-slave-1vw0"
server_key: "bb872310aa9be222843e900c3eb0c70f"
server_key_iv: "debe5c9835b5abd49deb97c78169d788"
server_key_version: "encryptionkey@0"
I20260928 15:51:00.571786 27937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:51:00.575632 27951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:00.577291 27937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:51:00.577577 27937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "863ddae0b0a54a8fbe6582c2d9183473"
format_stamp: "Formatted at 2026-09-28 15:51:00 on dist-test-slave-1vw0"
server_key: "bb872310aa9be222843e900c3eb0c70f"
server_key_iv: "debe5c9835b5abd49deb97c78169d788"
server_key_version: "encryptionkey@0"
I20260928 15:51:00.577839 27937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:00.584204 27937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:00.585237 27937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:00.585554 27937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:00.586578 27937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:00.588096 27937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:00.588236 27937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:00.588405 27937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:00.588512 27937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:00.625556 27937 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:37075
I20260928 15:51:00.625661 28064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:37075 every 8 connection(s)
I20260928 15:51:00.627049 27937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:00.634647 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 27937
I20260928 15:51:00.634865 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:51:00.635526 23159 external_mini_cluster.cc:1428] Setting key 91ad093a80b1c808ae14ba26149aed25
I20260928 15:51:00.636219 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:00.629863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:45243 (local address 127.22.157.254:35615)
0928 15:51:00.630357 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:00.630362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:00.631439 (+ 1077us) server_negotiation.cc:407] Connection header received
0928 15:51:00.633384 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:00.633391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:00.633464 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:00.633611 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:00.635251 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:00.635256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:00.635262 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:00.635283 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:00.635335 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:00.635347 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:00.635493 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:00.635813 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 15:51:00.635901 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":273,"threads_started":1}
I20260928 15:51:00.637488 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:00.630292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35615 (local address 127.22.157.193:45243)
0928 15:51:00.631253 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:00.631292 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:00.633078 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:00.633866 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:00.633875 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:00.633979 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:00.634867 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:00.634870 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:00.634947 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:00.635065 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:00.635547 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:00.635558 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:00.635569 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:51:00.635905 (+ 336us) client_negotiation.cc:240] Negotiation successful
0928 15:51:00.636181 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":289,"threads_started":1}
I20260928 15:51:00.638557 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35615
--builtin_ntp_servers=127.22.157.212:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 15:51:00.639119 28065 heartbeater.cc:347] Connected to a master server at 127.22.157.254:35615
I20260928 15:51:00.639424 28065 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:00.640017 28065 heartbeater.cc:511] Master 127.22.157.254:35615 requested a full tablet report, sending...
I20260928 15:51:00.642062 27878 ts_manager.cc:194] Registered new tserver with Master: 863ddae0b0a54a8fbe6582c2d9183473 (127.22.157.193:37075)
W20260928 15:51:00.974238 28069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:00.974587 28069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:00.974746 28069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:00.982865 28069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:00.982988 28069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:00.983204 28069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:51:00.993379 28069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:00.994884 28069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:00.996922 28069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:01.011456 28074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:01.011554 28075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:01.011469 28077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:01.012358 28069 server_base.cc:1034] running on GCE node
I20260928 15:51:01.013177 28069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:01.014259 28069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:01.015565 28069 hybrid_clock.cc:648] HybridClock initialized: now 1790610661015502 us; error 42 us; skew 500 ppm
I20260928 15:51:01.015997 28069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:01.018692 28069 webserver.cc:460] Webserver started at http://127.22.157.194:43903/ using document root <none> and password file <none>
I20260928 15:51:01.019451 28069 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:01.019593 28069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:01.019949 28069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:01.023056 28069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "d5117d816e484dfe99ccc0ba54be249a"
format_stamp: "Formatted at 2026-09-28 15:51:01 on dist-test-slave-1vw0"
server_key: "a8cab043c4a79af9eb17cb662afb565c"
server_key_iv: "ed11420f9f3a6bb16641e490532fc8e4"
server_key_version: "encryptionkey@0"
I20260928 15:51:01.023906 28069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "d5117d816e484dfe99ccc0ba54be249a"
format_stamp: "Formatted at 2026-09-28 15:51:01 on dist-test-slave-1vw0"
server_key: "a8cab043c4a79af9eb17cb662afb565c"
server_key_iv: "ed11420f9f3a6bb16641e490532fc8e4"
server_key_version: "encryptionkey@0"
I20260928 15:51:01.029584 28069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:51:01.033450 28083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:01.035054 28069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 15:51:01.035240 28069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "d5117d816e484dfe99ccc0ba54be249a"
format_stamp: "Formatted at 2026-09-28 15:51:01 on dist-test-slave-1vw0"
server_key: "a8cab043c4a79af9eb17cb662afb565c"
server_key_iv: "ed11420f9f3a6bb16641e490532fc8e4"
server_key_version: "encryptionkey@0"
I20260928 15:51:01.035528 28069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:01.041531 28069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:01.042375 28069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:01.042659 28069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:01.043587 28069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:01.045212 28069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:01.045337 28069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:01.045483 28069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:01.045575 28069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:01.081130 28069 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:34387
I20260928 15:51:01.081169 28196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:34387 every 8 connection(s)
I20260928 15:51:01.082566 28069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:51:01.091194 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 28069
I20260928 15:51:01.091361 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.085417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:35899 (local address 127.22.157.254:35615)
0928 15:51:01.085665 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.085671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.087181 (+ 1510us) server_negotiation.cc:407] Connection header received
0928 15:51:01.088902 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.088907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.088989 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.089095 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.090547 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.090552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.090555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.090573 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.090608 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.090619 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.090803 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.090964 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.091046 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 15:51:01.091387 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:51:01.091908 23159 external_mini_cluster.cc:1428] Setting key 82e09a69ee8db0d3c13de14c00d17c76
I20260928 15:51:01.092373 28199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.086054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35615 (local address 127.22.157.194:35899)
0928 15:51:01.087004 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:01.087048 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:01.088565 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:01.089306 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:01.089315 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:01.089413 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:01.090197 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:01.090199 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:01.090240 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:01.090336 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:01.090761 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:01.090768 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:01.090773 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:51:01.090992 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 15:51:01.091209 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":276,"threads_started":1}
I20260928 15:51:01.093935 28197 heartbeater.cc:347] Connected to a master server at 127.22.157.254:35615
I20260928 15:51:01.094262 28197 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:01.094856 28197 heartbeater.cc:511] Master 127.22.157.254:35615 requested a full tablet report, sending...
I20260928 15:51:01.094877 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35615
--builtin_ntp_servers=127.22.157.212:34491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 15:51:01.096324 27878 ts_manager.cc:194] Registered new tserver with Master: d5117d816e484dfe99ccc0ba54be249a (127.22.157.194:34387)
W20260928 15:51:01.427021 28200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:01.427331 28200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:01.427484 28200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:01.435029 28200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:01.435150 28200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:01.435333 28200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:51:01.444904 28200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:01.446383 28200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:01.448204 28200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:01.462651 28206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:01.462690 28205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:01.462733 28208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:01.463446 28200 server_base.cc:1034] running on GCE node
I20260928 15:51:01.464450 28200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:01.465655 28200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:01.466879 28200 hybrid_clock.cc:648] HybridClock initialized: now 1790610661466793 us; error 50 us; skew 500 ppm
I20260928 15:51:01.467235 28200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:01.469867 28200 webserver.cc:460] Webserver started at http://127.22.157.195:37913/ using document root <none> and password file <none>
I20260928 15:51:01.470620 28200 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:01.470717 28200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:01.471027 28200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:01.474509 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "c0f8399f9ea54077a836168b05bcd6f2"
format_stamp: "Formatted at 2026-09-28 15:51:01 on dist-test-slave-1vw0"
server_key: "85e5d7591e98808f985cc832def8aa25"
server_key_iv: "187805295e1ff9ad2530e47bbc1c4617"
server_key_version: "encryptionkey@0"
I20260928 15:51:01.475360 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "c0f8399f9ea54077a836168b05bcd6f2"
format_stamp: "Formatted at 2026-09-28 15:51:01 on dist-test-slave-1vw0"
server_key: "85e5d7591e98808f985cc832def8aa25"
server_key_iv: "187805295e1ff9ad2530e47bbc1c4617"
server_key_version: "encryptionkey@0"
I20260928 15:51:01.481021 28200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:51:01.485219 28214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:01.486975 28200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:51:01.487205 28200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "c0f8399f9ea54077a836168b05bcd6f2"
format_stamp: "Formatted at 2026-09-28 15:51:01 on dist-test-slave-1vw0"
server_key: "85e5d7591e98808f985cc832def8aa25"
server_key_iv: "187805295e1ff9ad2530e47bbc1c4617"
server_key_version: "encryptionkey@0"
I20260928 15:51:01.487509 28200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:01.493781 28200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:01.494746 28200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:01.495076 28200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:01.496155 28200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:01.497797 28200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:01.497897 28200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:01.498031 28200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:01.498121 28200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:01.533316 28200 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:37825
I20260928 15:51:01.533413 28327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:37825 every 8 connection(s)
I20260928 15:51:01.534783 28200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:51:01.537845 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 28200
I20260928 15:51:01.538009 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:51:01.538424 23159 external_mini_cluster.cc:1428] Setting key afcffd7334b2aaa5b276e218f4d2800f
I20260928 15:51:01.544219 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.537826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:32909 (local address 127.22.157.254:35615)
0928 15:51:01.538063 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.538069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.539819 (+ 1750us) server_negotiation.cc:407] Connection header received
0928 15:51:01.542045 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.542050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.542127 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.542229 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.543535 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.543543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.543547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.543574 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.543611 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.543625 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.543736 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.543959 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.544014 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 15:51:01.545257 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.538463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35615 (local address 127.22.157.195:32909)
0928 15:51:01.539666 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:01.539716 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:01.541706 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:01.542438 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:01.542446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:01.542543 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:01.543218 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:01.543220 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:01.543260 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:01.543359 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:01.543719 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:01.543726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:01.543732 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:51:01.543948 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 15:51:01.544172 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":357,"threads_started":1}
I20260928 15:51:01.546820 28328 heartbeater.cc:347] Connected to a master server at 127.22.157.254:35615
I20260928 15:51:01.547147 28328 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:01.547742 28328 heartbeater.cc:511] Master 127.22.157.254:35615 requested a full tablet report, sending...
I20260928 15:51:01.549319 27878 ts_manager.cc:194] Registered new tserver with Master: c0f8399f9ea54077a836168b05bcd6f2 (127.22.157.195:37825)
I20260928 15:51:01.554173 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:51:01.561961 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.558943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.22.157.254:35615)
0928 15:51:01.559172 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.559181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.559847 (+ 666us) server_negotiation.cc:407] Connection header received
0928 15:51:01.560031 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.560035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.560091 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.560181 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.561231 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.561236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.561239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.561255 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.561289 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.561301 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.561452 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.561551 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.561603 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:51:01.566542 27878 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:51:01.574074 27878 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:51:01.596150 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.588525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.22.157.194:34387)
0928 15:51:01.589603 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.589614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.590942 (+ 1328us) server_negotiation.cc:407] Connection header received
0928 15:51:01.591182 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.591189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.591482 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.591624 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.595010 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.595019 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.595037 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.595067 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.595136 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.595161 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.595284 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.595668 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.595782 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":762,"threads_started":1}
I20260928 15:51:01.597515 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.587883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:37825 (local address 127.0.0.1:53552)
0928 15:51:01.590057 (+ 2174us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:01.590109 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:01.590265 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:01.592811 (+ 2546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:01.592819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:01.592944 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:01.593874 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:01.593878 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:01.593910 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:01.594005 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:01.596641 (+ 2636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:01.596645 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:01.596650 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:51:01.596793 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 15:51:01.596899 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1906,"thread_start_us":2781,"threads_started":1}
I20260928 15:51:01.597793 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.588335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.22.157.195:37825)
0928 15:51:01.590796 (+ 2461us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.590806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.590844 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:51:01.590950 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.590959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.592553 (+ 1594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.593653 (+ 1100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.596353 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.596366 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.596381 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.596403 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.596476 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.596502 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.597111 (+ 609us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.597186 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.597289 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2288,"thread_start_us":2117,"threads_started":1}
I20260928 15:51:01.598217 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.588111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:34387 (local address 127.0.0.1:41540)
0928 15:51:01.590821 (+ 2710us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:01.590839 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:01.591027 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:01.593243 (+ 2216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:01.593249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:01.593276 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:01.594753 (+ 1477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:01.594758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:01.594776 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:01.594823 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:01.595429 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:01.595442 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:01.595451 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 15:51:01.596755 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0928 15:51:01.596914 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2560,"mutex_wait_us":44,"thread_start_us":1530,"threads_started":1}
I20260928 15:51:01.601949 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.590661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37075 (local address 127.0.0.1:50744)
0928 15:51:01.598534 (+ 7873us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:01.598554 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:01.598646 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:01.599377 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:01.599381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:01.599417 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:01.600911 (+ 1494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:01.600916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:01.600936 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:01.600985 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:01.601432 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:01.601437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:01.601442 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:51:01.601569 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 15:51:01.601626 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7766,"mutex_wait_us":214,"thread_start_us":7871,"threads_started":1}
I20260928 15:51:01.602108 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.591080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.22.157.193:37075)
0928 15:51:01.598569 (+ 7489us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.598581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.598615 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:51:01.598913 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.598924 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.599180 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.599376 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.601166 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.601175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.601188 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.601217 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.601278 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.601299 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.601453 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.601610 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.601776 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7307,"thread_start_us":7836,"threads_started":1}
I20260928 15:51:01.602666 28131 tablet_service.cc:1467] Processing CreateTablet for tablet 9c209f65b655463fb8c11926a29d993c (DEFAULT_TABLE table=test-table [id=ea8a480e124a4553bb4d5c76edf22dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:01.602654 28262 tablet_service.cc:1467] Processing CreateTablet for tablet 9c209f65b655463fb8c11926a29d993c (DEFAULT_TABLE table=test-table [id=ea8a480e124a4553bb4d5c76edf22dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:01.604082 28262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c209f65b655463fb8c11926a29d993c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:01.604319 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c209f65b655463fb8c11926a29d993c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:01.606401 27999 tablet_service.cc:1467] Processing CreateTablet for tablet 9c209f65b655463fb8c11926a29d993c (DEFAULT_TABLE table=test-table [id=ea8a480e124a4553bb4d5c76edf22dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:01.607908 27999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c209f65b655463fb8c11926a29d993c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:01.616829 28346 tablet_bootstrap.cc:492] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: Bootstrap starting.
I20260928 15:51:01.620797 28346 tablet_bootstrap.cc:654] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:01.621167 28347 tablet_bootstrap.cc:492] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: Bootstrap starting.
I20260928 15:51:01.622442 28346 log.cc:824] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:01.624241 28349 tablet_bootstrap.cc:492] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473: Bootstrap starting.
I20260928 15:51:01.625483 28347 tablet_bootstrap.cc:654] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:01.626202 28346 tablet_bootstrap.cc:492] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: No bootstrap required, opened a new log
I20260928 15:51:01.626515 28346 ts_tablet_manager.cc:1397] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 15:51:01.626899 28347 log.cc:824] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:01.629428 28349 tablet_bootstrap.cc:654] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:01.631404 28349 log.cc:824] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:01.636945 28347 tablet_bootstrap.cc:492] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: No bootstrap required, opened a new log
I20260928 15:51:01.636951 28349 tablet_bootstrap.cc:492] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473: No bootstrap required, opened a new log
I20260928 15:51:01.637296 28349 ts_tablet_manager.cc:1397] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260928 15:51:01.637400 28347 ts_tablet_manager.cc:1397] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260928 15:51:01.637646 28346 raft_consensus.cc:348] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.638304 28346 raft_consensus.cc:374] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:01.638459 28346 raft_consensus.cc:729] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f8399f9ea54077a836168b05bcd6f2, State: Initialized, Role: FOLLOWER
I20260928 15:51:01.639333 28346 consensus_queue.cc:260] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.641255 28328 heartbeater.cc:503] Master 127.22.157.254:35615 was elected leader, sending a full tablet report...
I20260928 15:51:01.643374 28346 ts_tablet_manager.cc:1428] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260928 15:51:01.645053 28065 heartbeater.cc:503] Master 127.22.157.254:35615 was elected leader, sending a full tablet report...
I20260928 15:51:01.648814 28347 raft_consensus.cc:348] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.649334 28347 raft_consensus.cc:374] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:01.649447 28347 raft_consensus.cc:729] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5117d816e484dfe99ccc0ba54be249a, State: Initialized, Role: FOLLOWER
I20260928 15:51:01.650153 28347 consensus_queue.cc:260] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.650214 28349 raft_consensus.cc:348] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.650640 28349 raft_consensus.cc:374] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:01.650782 28349 raft_consensus.cc:729] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863ddae0b0a54a8fbe6582c2d9183473, State: Initialized, Role: FOLLOWER
I20260928 15:51:01.651433 28197 heartbeater.cc:503] Master 127.22.157.254:35615 was elected leader, sending a full tablet report...
I20260928 15:51:01.651427 28349 consensus_queue.cc:260] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.652247 28347 ts_tablet_manager.cc:1428] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 15:51:01.653887 28349 ts_tablet_manager.cc:1428] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260928 15:51:01.766037 28354 raft_consensus.cc:482] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:01.766418 28354 raft_consensus.cc:504] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.768296 28354 leader_election.cc:288] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f8399f9ea54077a836168b05bcd6f2 (127.22.157.195:37825), 863ddae0b0a54a8fbe6582c2d9183473 (127.22.157.193:37075)
I20260928 15:51:01.772084 28355 raft_consensus.cc:482] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:01.772421 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.768874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:39303 (local address 127.22.157.195:37825)
0928 15:51:01.769109 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.769116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.769625 (+ 509us) server_negotiation.cc:407] Connection header received
0928 15:51:01.769935 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.769944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.770018 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.770130 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.771526 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.771532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.771537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.771558 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.771623 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.771647 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.771818 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.772062 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.772140 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:51:01.772557 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.769462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:43891 (local address 127.22.157.193:37075)
0928 15:51:01.769684 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.769691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.770024 (+ 333us) server_negotiation.cc:407] Connection header received
0928 15:51:01.770192 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.770197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.770261 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.770393 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.771774 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.771780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.771784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.771806 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.771849 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.771865 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.772020 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.772137 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.772219 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:51:01.772773 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.769085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37075 (local address 127.22.157.194:43891)
0928 15:51:01.769896 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:01.769913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:01.770039 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:01.770460 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:01.770464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:01.770529 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:01.771509 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:01.771512 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:01.771529 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:01.771571 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:01.771995 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:01.772001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:01.772005 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 15:51:01.772163 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 15:51:01.772219 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":418,"threads_started":1}
I20260928 15:51:01.772472 28355 raft_consensus.cc:504] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.773262 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.768499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:37825 (local address 127.22.157.194:39303)
0928 15:51:01.769474 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:01.769506 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:01.769748 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:01.770197 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:01.770202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:01.770225 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:01.771273 (+ 1048us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:01.771276 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:01.771292 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:01.771339 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:01.771896 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:01.771901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:01.771906 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:51:01.772768 (+ 862us) client_negotiation.cc:240] Negotiation successful
0928 15:51:01.772830 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":552,"threads_started":1}
I20260928 15:51:01.773916 28020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c209f65b655463fb8c11926a29d993c" candidate_uuid: "d5117d816e484dfe99ccc0ba54be249a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863ddae0b0a54a8fbe6582c2d9183473" is_pre_election: true
I20260928 15:51:01.774466 28020 raft_consensus.cc:2454] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5117d816e484dfe99ccc0ba54be249a in term 0.
I20260928 15:51:01.774573 28282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c209f65b655463fb8c11926a29d993c" candidate_uuid: "d5117d816e484dfe99ccc0ba54be249a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f8399f9ea54077a836168b05bcd6f2" is_pre_election: true
I20260928 15:51:01.775143 28282 raft_consensus.cc:2454] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5117d816e484dfe99ccc0ba54be249a in term 0.
I20260928 15:51:01.775424 28085 leader_election.cc:302] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 863ddae0b0a54a8fbe6582c2d9183473, d5117d816e484dfe99ccc0ba54be249a; no voters:
I20260928 15:51:01.775681 28355 leader_election.cc:288] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0f8399f9ea54077a836168b05bcd6f2 (127.22.157.195:37825), d5117d816e484dfe99ccc0ba54be249a (127.22.157.194:34387)
I20260928 15:51:01.775872 28354 raft_consensus.cc:2781] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:51:01.776068 28354 raft_consensus.cc:482] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:01.776206 28354 raft_consensus.cc:3037] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:01.779398 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.775502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:48879 (local address 127.22.157.195:37825)
0928 15:51:01.775691 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.775699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.776903 (+ 1204us) server_negotiation.cc:407] Connection header received
0928 15:51:01.777129 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.777134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.777208 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.777333 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.778678 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.778684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.778689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.778714 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.778747 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.778760 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.778923 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.778977 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.779058 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:51:01.779481 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.775066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:37825 (local address 127.22.157.193:48879)
0928 15:51:01.776778 (+ 1712us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:01.776805 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:01.777002 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:01.777416 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:01.777421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:01.777458 (+ 37us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:01.778432 (+ 974us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:01.778435 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:01.778449 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:01.778491 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:01.778875 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:01.778880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:01.778885 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:51:01.779013 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 15:51:01.779067 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"thread_start_us":1782,"threads_started":1}
I20260928 15:51:01.780632 28282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c209f65b655463fb8c11926a29d993c" candidate_uuid: "863ddae0b0a54a8fbe6582c2d9183473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f8399f9ea54077a836168b05bcd6f2" is_pre_election: true
I20260928 15:51:01.780943 28282 raft_consensus.cc:2454] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 863ddae0b0a54a8fbe6582c2d9183473 in term 0.
I20260928 15:51:01.780948 28354 raft_consensus.cc:504] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.781840 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.775788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:51263 (local address 127.22.157.194:34387)
0928 15:51:01.775943 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.775949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.777318 (+ 1369us) server_negotiation.cc:407] Connection header received
0928 15:51:01.777547 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.777553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.777621 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.777734 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.778927 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.778938 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.778943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.778962 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.779019 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.779035 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.779122 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.779364 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.779431 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:51:01.782251 28282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c209f65b655463fb8c11926a29d993c" candidate_uuid: "d5117d816e484dfe99ccc0ba54be249a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f8399f9ea54077a836168b05bcd6f2"
I20260928 15:51:01.782538 28282 raft_consensus.cc:3037] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:01.782802 27954 leader_election.cc:302] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 863ddae0b0a54a8fbe6582c2d9183473, c0f8399f9ea54077a836168b05bcd6f2; no voters:
I20260928 15:51:01.784035 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.775314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:34387 (local address 127.22.157.193:51263)
0928 15:51:01.777179 (+ 1865us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:01.777200 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:01.777356 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:01.777780 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:01.777784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:01.777810 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:01.778687 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:01.778690 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:01.778707 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:01.778752 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:01.779223 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:01.779230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:01.779237 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 15:51:01.783456 (+ 4219us) client_negotiation.cc:240] Negotiation successful
0928 15:51:01.783542 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"thread_start_us":1134,"threads_started":1}
I20260928 15:51:01.784596 28355 raft_consensus.cc:2781] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:51:01.784780 28355 raft_consensus.cc:482] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:01.784890 28355 raft_consensus.cc:3037] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:01.786149 28282 raft_consensus.cc:2454] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5117d816e484dfe99ccc0ba54be249a in term 1.
I20260928 15:51:01.786623 28086 leader_election.cc:302] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0f8399f9ea54077a836168b05bcd6f2, d5117d816e484dfe99ccc0ba54be249a; no voters:
I20260928 15:51:01.787592 28354 leader_election.cc:288] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [CANDIDATE]: Term 1 election: Requested vote from peers c0f8399f9ea54077a836168b05bcd6f2 (127.22.157.195:37825), 863ddae0b0a54a8fbe6582c2d9183473 (127.22.157.193:37075)
I20260928 15:51:01.788056 28355 raft_consensus.cc:504] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.788311 28360 raft_consensus.cc:2781] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:01.788589 28360 raft_consensus.cc:686] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 1 LEADER]: Becoming Leader. State: Replica: d5117d816e484dfe99ccc0ba54be249a, State: Running, Role: LEADER
I20260928 15:51:01.789302 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c209f65b655463fb8c11926a29d993c" candidate_uuid: "863ddae0b0a54a8fbe6582c2d9183473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5117d816e484dfe99ccc0ba54be249a" is_pre_election: true
I20260928 15:51:01.789351 28360 consensus_queue.cc:237] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } }
I20260928 15:51:01.790337 28020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c209f65b655463fb8c11926a29d993c" candidate_uuid: "d5117d816e484dfe99ccc0ba54be249a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863ddae0b0a54a8fbe6582c2d9183473"
W20260928 15:51:01.790558 28329 tablet.cc:2369] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:51:01.790635 28020 raft_consensus.cc:2379] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d5117d816e484dfe99ccc0ba54be249a in current term 1: Already voted for candidate 863ddae0b0a54a8fbe6582c2d9183473 in this term.
I20260928 15:51:01.791785 28355 leader_election.cc:288] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [CANDIDATE]: Term 1 election: Requested vote from peers c0f8399f9ea54077a836168b05bcd6f2 (127.22.157.195:37825), d5117d816e484dfe99ccc0ba54be249a (127.22.157.194:34387)
I20260928 15:51:01.792078 28282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c209f65b655463fb8c11926a29d993c" candidate_uuid: "863ddae0b0a54a8fbe6582c2d9183473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f8399f9ea54077a836168b05bcd6f2"
I20260928 15:51:01.792416 28282 raft_consensus.cc:2379] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 863ddae0b0a54a8fbe6582c2d9183473 in current term 1: Already voted for candidate d5117d816e484dfe99ccc0ba54be249a in this term.
I20260928 15:51:01.792375 28152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c209f65b655463fb8c11926a29d993c" candidate_uuid: "863ddae0b0a54a8fbe6582c2d9183473" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5117d816e484dfe99ccc0ba54be249a"
I20260928 15:51:01.793262 27954 leader_election.cc:302] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 863ddae0b0a54a8fbe6582c2d9183473; no voters: c0f8399f9ea54077a836168b05bcd6f2, d5117d816e484dfe99ccc0ba54be249a
I20260928 15:51:01.793668 28355 raft_consensus.cc:2726] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 15:51:01.795537 27878 catalog_manager.cc:5495] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a reported cstate change: term changed from 0 to 1, leader changed from <none> to d5117d816e484dfe99ccc0ba54be249a (127.22.157.194). New cstate: current_term: 1 leader_uuid: "d5117d816e484dfe99ccc0ba54be249a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5117d816e484dfe99ccc0ba54be249a" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 } health_report { overall_health: UNKNOWN } } }
W20260928 15:51:01.834843 28198 tablet.cc:2369] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:51:01.839766 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.837433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.22.157.194:34387)
0928 15:51:01.837645 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.837650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.837671 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:51:01.837763 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.837767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.837865 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.837985 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.838869 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.838873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.838877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.838893 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.838925 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.838968 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.839077 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.839279 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.839362 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:51:01.848466 28020 raft_consensus.cc:1261] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 1 FOLLOWER]: Refusing update from remote peer d5117d816e484dfe99ccc0ba54be249a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:01.848448 28282 raft_consensus.cc:1261] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 1 FOLLOWER]: Refusing update from remote peer d5117d816e484dfe99ccc0ba54be249a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:01.849432 28360 consensus_queue.cc:1035] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0f8399f9ea54077a836168b05bcd6f2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 37825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:01.849767 28354 consensus_queue.cc:1035] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [LEADER]: Connected to new peer: Peer: permanent_uuid: "863ddae0b0a54a8fbe6582c2d9183473" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:01.881671 27876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:01.881973 27876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:01.885813 27876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0d795832afd43c9a5e6c648247ca605: Sending DeleteTablet for 3 replicas of tablet 9c209f65b655463fb8c11926a29d993c
I20260928 15:51:01.887115 28262 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c209f65b655463fb8c11926a29d993c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:01 UTC) from {username='slave'} at 127.0.0.1:53552
I20260928 15:51:01.887239 28131 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c209f65b655463fb8c11926a29d993c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:01 UTC) from {username='slave'} at 127.0.0.1:41540
I20260928 15:51:01.888252 27999 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c209f65b655463fb8c11926a29d993c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:01 UTC) from {username='slave'} at 127.0.0.1:50744
I20260928 15:51:01.890748 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 27937
I20260928 15:51:01.897194 28375 tablet_replica.cc:330] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473: stopping tablet replica
I20260928 15:51:01.897850 28375 raft_consensus.cc:2229] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:01.898562 28375 raft_consensus.cc:2258] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:01.901162 28373 tablet_replica.cc:330] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: stopping tablet replica
I20260928 15:51:01.901770 28373 raft_consensus.cc:2229] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:01.902684 28373 raft_consensus.cc:2258] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:01.902918 28375 ts_tablet_manager.cc:1905] T 9c209f65b655463fb8c11926a29d993c P 863ddae0b0a54a8fbe6582c2d9183473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:01.903311 28374 tablet_replica.cc:330] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: stopping tablet replica
I20260928 15:51:01.903725 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:01.893917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.22.157.193:37075)
0928 15:51:01.894191 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:01.894201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:01.900761 (+ 6560us) server_negotiation.cc:407] Connection header received
0928 15:51:01.901089 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:01.901095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:01.901201 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:01.901307 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:01.902780 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:01.902786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:01.902791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:01.902814 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:01.902856 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:01.902873 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:01.902991 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:01.903346 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 15:51:01.903418 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:51:01.903766 28374 raft_consensus.cc:2229] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:51:01.904421 28374 raft_consensus.cc:2258] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:01.905866 28060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:01.910391 28373 ts_tablet_manager.cc:1905] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:01.911762 28374 ts_tablet_manager.cc:1905] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:01.919130 28373 ts_tablet_manager.cc:1918] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:01.919363 28373 log.cc:1196] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/9c209f65b655463fb8c11926a29d993c
I20260928 15:51:01.919610 28374 ts_tablet_manager.cc:1918] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:01.919898 28374 log.cc:1196] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/9c209f65b655463fb8c11926a29d993c
I20260928 15:51:01.920006 28373 ts_tablet_manager.cc:1939] T 9c209f65b655463fb8c11926a29d993c P c0f8399f9ea54077a836168b05bcd6f2: Deleting consensus metadata
I20260928 15:51:01.920586 28374 ts_tablet_manager.cc:1939] T 9c209f65b655463fb8c11926a29d993c P d5117d816e484dfe99ccc0ba54be249a: Deleting consensus metadata
I20260928 15:51:01.922475 27863 catalog_manager.cc:4841] TS d5117d816e484dfe99ccc0ba54be249a (127.22.157.194:34387): tablet 9c209f65b655463fb8c11926a29d993c (table test-table [id=ea8a480e124a4553bb4d5c76edf22dda]) successfully deleted
I20260928 15:51:01.922509 27864 catalog_manager.cc:4841] TS c0f8399f9ea54077a836168b05bcd6f2 (127.22.157.195:37825): tablet 9c209f65b655463fb8c11926a29d993c (table test-table [id=ea8a480e124a4553bb4d5c76edf22dda]) successfully deleted
I20260928 15:51:02.002345 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 27937
W20260928 15:51:02.020020 27863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.157.193:37075 (error 108)
I20260928 15:51:02.020838 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 28069
I20260928 15:51:02.022413 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.021809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37075 (local address 127.0.0.1:50748)
0928 15:51:02.022080 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:02.022214 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260928 15:51:02.022748 27863 catalog_manager.cc:4584] TS 863ddae0b0a54a8fbe6582c2d9183473 (127.22.157.193:37075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c209f65b655463fb8c11926a29d993c: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37075: connect: Connection refused (error 111)
I20260928 15:51:02.024052 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.021702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.22.157.194:34387)
0928 15:51:02.021898 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:02.021905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:02.022043 (+ 138us) server_negotiation.cc:407] Connection header received
0928 15:51:02.022265 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:02.022272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:02.022338 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:02.022459 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:02.023379 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:02.023385 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:02.023390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:02.023409 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:02.023455 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:02.023467 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:02.023611 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:02.023700 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 15:51:02.023765 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:51:02.025072 28192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:02.076396 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.075920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37075 (local address 127.0.0.1:50750)
0928 15:51:02.076176 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:02.076261 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 15:51:02.077492 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.077141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37075 (local address 127.0.0.1:50762)
0928 15:51:02.077296 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:02.077375 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:51:02.136211 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 28069
I20260928 15:51:02.138906 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.138350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37075 (local address 127.0.0.1:50778)
0928 15:51:02.138577 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:02.138701 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 15:51:02.141877 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.141403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37075 (local address 127.0.0.1:50790)
0928 15:51:02.141606 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:02.141707 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 15:51:02.155105 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 28200
I20260928 15:51:02.158223 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.155879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.22.157.195:37825)
0928 15:51:02.156108 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:02.156115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:02.156141 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:51:02.156374 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:02.156384 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:02.156463 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:02.156582 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:02.157599 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:02.157605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:02.157609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:02.157632 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:02.157665 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:02.157677 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:02.157807 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:02.157898 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 15:51:02.157962 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:51:02.158874 28322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:02.222363 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.221723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37075 (local address 127.0.0.1:50796)
0928 15:51:02.222021 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:02.222145 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 15:51:02.223716 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.223184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37075 (local address 127.0.0.1:50808)
0928 15:51:02.223431 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:02.223520 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 15:51:02.248430 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 28200
I20260928 15:51:02.265568 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 27846
I20260928 15:51:02.266975 27908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:02.375224 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 27846
2026-09-28T15:51:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2820 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 15:51:02.417825 23159 mini_oidc.cc:152] Starting JWKS server
I20260928 15:51:02.420400 23159 webserver.cc:460] Webserver started at https://127.0.0.1:41891/ using document root <none> and password file <none>
I20260928 15:51:02.420809 23159 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:51:02.421542 23159 webserver.cc:460] Webserver started at http://127.0.0.1:38665/ using document root <none> and password file <none>
2026-09-28T15:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:02Z Disabled control of system clock
I20260928 15:51:02.452198 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:35359
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:35359
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41891/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/testchainca.cert with env {}
W20260928 15:51:02.801337 28387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:02.801668 28387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:02.801759 28387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:02.809581 28387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:02.809718 28387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:02.809756 28387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:02.809808 28387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:02.809854 28387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:51:02.809916 28387 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:51:02.809957 28387 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41891/jwks.json
I20260928 15:51:02.820571 28387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:35359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:35359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41891/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:02.822156 28387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:02.823632 28387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:02.836359 28393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:02.836387 28395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:02.836350 28392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:02.837003 28387 server_base.cc:1034] running on GCE node
I20260928 15:51:02.837954 28387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:02.839388 28387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:02.840618 28387 hybrid_clock.cc:648] HybridClock initialized: now 1790610662840574 us; error 36 us; skew 500 ppm
I20260928 15:51:02.841038 28387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:02.843657 28387 webserver.cc:460] Webserver started at http://127.22.157.254:43801/ using document root <none> and password file <none>
I20260928 15:51:02.844327 28387 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:02.844420 28387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:02.844934 28387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:02.848146 28387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "622553c2bab940e08e0674aff501ac59"
format_stamp: "Formatted at 2026-09-28 15:51:02 on dist-test-slave-1vw0"
server_key: "bf030b6ee3cbc25d001c2081f853f2a7"
server_key_iv: "f70f099ff2d353d070aaad1a3fe2a016"
server_key_version: "encryptionkey@0"
I20260928 15:51:02.849102 28387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "622553c2bab940e08e0674aff501ac59"
format_stamp: "Formatted at 2026-09-28 15:51:02 on dist-test-slave-1vw0"
server_key: "bf030b6ee3cbc25d001c2081f853f2a7"
server_key_iv: "f70f099ff2d353d070aaad1a3fe2a016"
server_key_version: "encryptionkey@0"
I20260928 15:51:02.855453 28387 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 15:51:02.860044 28401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:02.861864 28387 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:51:02.862089 28387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "622553c2bab940e08e0674aff501ac59"
format_stamp: "Formatted at 2026-09-28 15:51:02 on dist-test-slave-1vw0"
server_key: "bf030b6ee3cbc25d001c2081f853f2a7"
server_key_iv: "f70f099ff2d353d070aaad1a3fe2a016"
server_key_version: "encryptionkey@0"
I20260928 15:51:02.862442 28387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:02.879328 28387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:02.880225 28387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:02.880570 28387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:02.900712 28387 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:35359
I20260928 15:51:02.900800 28453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:35359 every 8 connection(s)
I20260928 15:51:02.902406 28387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:02.905129 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 28387
I20260928 15:51:02.905350 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:02.906026 23159 external_mini_cluster.cc:1428] Setting key 95292144c9e1e8772a360aabd279d88d
I20260928 15:51:02.907063 28454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:02.919927 28457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:02.908084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.22.157.254:35359)
0928 15:51:02.909193 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:02.909205 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:02.909248 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:51:02.910809 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:02.910832 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:02.911385 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:02.911822 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:02.912436 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:02.913861 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:02.915322 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:02.916013 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:02.916221 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:02.917489 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:02.917503 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:02.917518 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:02.918128 (+ 610us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:02.918209 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:02.918231 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:02.918369 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:02.918618 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 15:51:02.918946 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":392,"threads_started":1}
I20260928 15:51:02.920035 28454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59: Bootstrap starting.
I20260928 15:51:02.923775 28454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:02.924247 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:51:02.925017 28454 log.cc:824] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:02.928148 28454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59: No bootstrap required, opened a new log
I20260928 15:51:02.935935 28454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622553c2bab940e08e0674aff501ac59" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35359 } }
I20260928 15:51:02.936403 28454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:02.936522 28454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622553c2bab940e08e0674aff501ac59, State: Initialized, Role: FOLLOWER
I20260928 15:51:02.937186 28454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622553c2bab940e08e0674aff501ac59" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35359 } }
I20260928 15:51:02.937448 28454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:02.937597 28454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:02.937781 28454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:02.939996 28454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622553c2bab940e08e0674aff501ac59" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35359 } }
I20260928 15:51:02.940610 28454 leader_election.cc:302] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 622553c2bab940e08e0674aff501ac59; no voters:
I20260928 15:51:02.941269 28454 leader_election.cc:288] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:02.941483 28459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:02.942314 28459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [term 1 LEADER]: Becoming Leader. State: Replica: 622553c2bab940e08e0674aff501ac59, State: Running, Role: LEADER
I20260928 15:51:02.942953 28459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622553c2bab940e08e0674aff501ac59" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35359 } }
I20260928 15:51:02.943712 28454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:02.945811 28461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 622553c2bab940e08e0674aff501ac59. Latest consensus state: current_term: 1 leader_uuid: "622553c2bab940e08e0674aff501ac59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622553c2bab940e08e0674aff501ac59" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35359 } } }
I20260928 15:51:02.946292 28461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:02.946589 28460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "622553c2bab940e08e0674aff501ac59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "622553c2bab940e08e0674aff501ac59" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35359 } } }
I20260928 15:51:02.947288 28460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:02.955227 28464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:02.961107 28464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:02.970873 28464 catalog_manager.cc:1347] Generated new cluster ID: 3775805b03394f86b86ed3d825fa3402
I20260928 15:51:02.971024 28464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:02.989816 28464 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:02.991228 28464 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:03.002058 28464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 622553c2bab940e08e0674aff501ac59: Generated new TSK 0
I20260928 15:51:03.003415 28464 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 15:51:03.045009 28484 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 15:51:03.045359 28484 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:51:03.046422 28457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:03.022628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.22.157.254:35359)
0928 15:51:03.022853 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:03.022860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:03.023428 (+ 568us) server_negotiation.cc:407] Connection header received
0928 15:51:03.023710 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:03.023716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:03.023821 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:03.023997 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:51:03.024540 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.025718 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.027329 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.028219 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.028377 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:03.029191 (+ 814us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:51:03.045632 (+ 16441us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 15:51:03.046074 (+ 442us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34486: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 15:51:03.046558 28457 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34486: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 15:51:03.047915 28483 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:03.022611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35359 (local address 127.0.0.1:34486)
0928 15:51:03.023313 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:03.023331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:03.023485 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:03.024025 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:03.024029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:03.024040 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:51:03.024389 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:03.024397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.025918 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.025923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:03.027159 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:03.027174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.028456 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.028463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:03.028609 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:03.028683 (+ 74us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:51:03.046023 (+ 17340us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 15:51:03.046333 (+ 310us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.157.254:35359: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":359,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.157.254:35359: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 15:51:03.049476 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 28387
I20260928 15:51:03.050717 28449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:03.145227 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 28387
2026-09-28T15:51:03Z chronyd exiting
I20260928 15:51:03.196614 23159 test_util.cc:175] -----------------------------------------------
I20260928 15:51:03.196833 23159 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790610618730370-23159-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (783 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 15:51:03.200613 23159 mini_oidc.cc:152] Starting JWKS server
I20260928 15:51:03.202677 23159 webserver.cc:460] Webserver started at https://127.0.0.1:35129/ using document root <none> and password file <none>
I20260928 15:51:03.202943 23159 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:51:03.203919 23159 webserver.cc:460] Webserver started at http://127.0.0.1:34953/ using document root <none> and password file <none>
2026-09-28T15:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:03Z Disabled control of system clock
I20260928 15:51:03.235532 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45505
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:45505
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35129/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/testchainca.cert with env {}
W20260928 15:51:03.577385 28491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:03.577782 28491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:03.577961 28491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:03.585439 28491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:03.585618 28491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:03.585724 28491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:03.585845 28491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:03.585937 28491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:51:03.586032 28491 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:51:03.586112 28491 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35129/jwks.json
I20260928 15:51:03.596232 28491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:45505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35129/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:03.597697 28491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:03.599109 28491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:03.612149 28497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:03.612098 28496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:03.612152 28499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:03.613062 28491 server_base.cc:1034] running on GCE node
I20260928 15:51:03.613967 28491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:03.615542 28491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:03.616832 28491 hybrid_clock.cc:648] HybridClock initialized: now 1790610663616776 us; error 51 us; skew 500 ppm
I20260928 15:51:03.617246 28491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:03.619623 28491 webserver.cc:460] Webserver started at http://127.22.157.254:33313/ using document root <none> and password file <none>
I20260928 15:51:03.620397 28491 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:03.620539 28491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:03.620944 28491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:03.623931 28491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "669e47e4e80b4519986f5735f0ec5a87"
format_stamp: "Formatted at 2026-09-28 15:51:03 on dist-test-slave-1vw0"
server_key: "bc45190fe1178fd32a369fb68a97ce21"
server_key_iv: "edddd583999dc3aeb7289dff34243396"
server_key_version: "encryptionkey@0"
I20260928 15:51:03.624874 28491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "669e47e4e80b4519986f5735f0ec5a87"
format_stamp: "Formatted at 2026-09-28 15:51:03 on dist-test-slave-1vw0"
server_key: "bc45190fe1178fd32a369fb68a97ce21"
server_key_iv: "edddd583999dc3aeb7289dff34243396"
server_key_version: "encryptionkey@0"
I20260928 15:51:03.630946 28491 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:51:03.634948 28506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:03.636663 28491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:51:03.636895 28491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "669e47e4e80b4519986f5735f0ec5a87"
format_stamp: "Formatted at 2026-09-28 15:51:03 on dist-test-slave-1vw0"
server_key: "bc45190fe1178fd32a369fb68a97ce21"
server_key_iv: "edddd583999dc3aeb7289dff34243396"
server_key_version: "encryptionkey@0"
I20260928 15:51:03.637113 28491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:03.654281 28491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:03.655196 28491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:03.655483 28491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:03.674831 28491 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:45505
I20260928 15:51:03.674857 28557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:45505 every 8 connection(s)
I20260928 15:51:03.676285 28491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:03.677968 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 28491
I20260928 15:51:03.678189 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:03.678853 23159 external_mini_cluster.cc:1428] Setting key 966f3325cb3da5f9001cb59ca0bde40b
I20260928 15:51:03.680967 28558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:03.696286 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:03.680875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59158 (local address 127.22.157.254:45505)
0928 15:51:03.682076 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:03.682091 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:03.682146 (+ 55us) server_negotiation.cc:407] Connection header received
0928 15:51:03.684344 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:03.684375 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:03.685201 (+ 826us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:03.685772 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:03.686407 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.688629 (+ 2222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.690346 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.691478 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.691706 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:03.693079 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:03.693090 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:03.693117 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:03.693949 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:03.694066 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:03.694126 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:03.694248 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:03.694486 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 15:51:03.694853 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":411,"threads_started":1}
I20260928 15:51:03.696815 28558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87: Bootstrap starting.
I20260928 15:51:03.700286 28558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:03.700611 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:51:03.701716 28558 log.cc:824] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:03.704993 28558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87: No bootstrap required, opened a new log
I20260928 15:51:03.712312 28558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669e47e4e80b4519986f5735f0ec5a87" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45505 } }
I20260928 15:51:03.712808 28558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:03.712920 28558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669e47e4e80b4519986f5735f0ec5a87, State: Initialized, Role: FOLLOWER
I20260928 15:51:03.713475 28558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669e47e4e80b4519986f5735f0ec5a87" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45505 } }
I20260928 15:51:03.713701 28558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:03.713797 28558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:03.713917 28558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:03.716557 28558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669e47e4e80b4519986f5735f0ec5a87" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45505 } }
I20260928 15:51:03.717372 28558 leader_election.cc:302] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 669e47e4e80b4519986f5735f0ec5a87; no voters:
I20260928 15:51:03.718148 28558 leader_election.cc:288] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:03.718304 28563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:03.719113 28563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [term 1 LEADER]: Becoming Leader. State: Replica: 669e47e4e80b4519986f5735f0ec5a87, State: Running, Role: LEADER
I20260928 15:51:03.719712 28563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669e47e4e80b4519986f5735f0ec5a87" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45505 } }
I20260928 15:51:03.720702 28558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:03.723167 28564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "669e47e4e80b4519986f5735f0ec5a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669e47e4e80b4519986f5735f0ec5a87" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45505 } } }
I20260928 15:51:03.723556 28564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:03.724062 28565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 669e47e4e80b4519986f5735f0ec5a87. Latest consensus state: current_term: 1 leader_uuid: "669e47e4e80b4519986f5735f0ec5a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669e47e4e80b4519986f5735f0ec5a87" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45505 } } }
I20260928 15:51:03.724392 28565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:03.725865 28568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:03.732343 28568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:03.745015 28568 catalog_manager.cc:1347] Generated new cluster ID: 37da907d97124fb6b43a72c946132628
I20260928 15:51:03.745177 28568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:03.755748 28568 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:03.757769 28568 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:03.765980 28568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 669e47e4e80b4519986f5735f0ec5a87: Generated new TSK 0
I20260928 15:51:03.767372 28568 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 15:51:03.811976 28588 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 15:51:03.812146 28588 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:51:03.813745 28561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:03.787684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.22.157.254:45505)
0928 15:51:03.787904 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:03.787911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:03.789542 (+ 1631us) server_negotiation.cc:407] Connection header received
0928 15:51:03.789778 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:03.789784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:03.789897 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:03.790289 (+ 392us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:51:03.791262 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.792453 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.794163 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.794860 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.795016 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:03.795771 (+ 755us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:51:03.812681 (+ 16910us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 15:51:03.813190 (+ 509us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59168: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 15:51:03.813916 28561 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59168: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 15:51:03.813951 28587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:03.787468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45505 (local address 127.0.0.1:59168)
0928 15:51:03.788818 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:03.788842 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:03.789324 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:03.790562 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:03.790568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:03.790605 (+ 37us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:51:03.791023 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:03.791041 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.792670 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.792675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:03.793948 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:03.793965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:03.795120 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:03.795126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:03.795306 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:03.795348 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:51:03.813060 (+ 17712us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 15:51:03.813303 (+ 243us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.157.254:45505: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":734,"threads_started":1}
I20260928 15:51:03.815840 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 28491
I20260928 15:51:03.817157 28553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:03.916600 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 28491
2026-09-28T15:51:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (772 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 15:51:03.974074 23159 mini_oidc.cc:152] Starting JWKS server
I20260928 15:51:03.976184 23159 webserver.cc:460] Webserver started at https://127.0.0.1:39495/ using document root <none> and password file <none>
I20260928 15:51:03.976491 23159 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 15:51:03.977231 23159 webserver.cc:460] Webserver started at http://127.0.0.1:36497/ using document root <none> and password file <none>
2026-09-28T15:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:03Z Disabled control of system clock
I20260928 15:51:04.009683 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:34289
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:34289
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39495/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 15:51:04.346079 28595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:04.346521 28595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:04.346694 28595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:04.354491 28595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:04.354688 28595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:04.354769 28595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:04.354861 28595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:04.354952 28595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:51:04.355053 28595 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:51:04.355131 28595 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39495/jwks.json
I20260928 15:51:04.364917 28595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:34289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:34289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39495/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:04.366411 28595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:04.368721 28595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:04.381748 28600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:04.381768 28601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:04.382575 28603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:04.382819 28595 server_base.cc:1034] running on GCE node
I20260928 15:51:04.383736 28595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:04.385360 28595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:04.386616 28595 hybrid_clock.cc:648] HybridClock initialized: now 1790610664386594 us; error 78 us; skew 500 ppm
I20260928 15:51:04.386984 28595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:04.397186 28595 webserver.cc:460] Webserver started at http://127.22.157.254:38521/ using document root <none> and password file <none>
I20260928 15:51:04.398005 28595 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:04.398152 28595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:04.398505 28595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:04.401659 28595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "f40ef4ead452488981b46314057fc1aa"
format_stamp: "Formatted at 2026-09-28 15:51:04 on dist-test-slave-1vw0"
server_key: "fc2328fb0d2f7d5efe20f6b388478790"
server_key_iv: "b6d0f13ba2ded55d33ed801f606de480"
server_key_version: "encryptionkey@0"
I20260928 15:51:04.402499 28595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "f40ef4ead452488981b46314057fc1aa"
format_stamp: "Formatted at 2026-09-28 15:51:04 on dist-test-slave-1vw0"
server_key: "fc2328fb0d2f7d5efe20f6b388478790"
server_key_iv: "b6d0f13ba2ded55d33ed801f606de480"
server_key_version: "encryptionkey@0"
I20260928 15:51:04.407929 28595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:51:04.411552 28609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:04.413172 28595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:04.413364 28595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "f40ef4ead452488981b46314057fc1aa"
format_stamp: "Formatted at 2026-09-28 15:51:04 on dist-test-slave-1vw0"
server_key: "fc2328fb0d2f7d5efe20f6b388478790"
server_key_iv: "b6d0f13ba2ded55d33ed801f606de480"
server_key_version: "encryptionkey@0"
I20260928 15:51:04.413584 28595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:04.443300 28595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:04.444236 28595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:04.444579 28595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:04.465413 28595 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:34289
I20260928 15:51:04.465420 28661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:34289 every 8 connection(s)
I20260928 15:51:04.466995 28595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:04.471060 28662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:04.473910 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 28595
I20260928 15:51:04.474152 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:04.474946 23159 external_mini_cluster.cc:1428] Setting key d60902d127055774d40adc99a26dadba
I20260928 15:51:04.483817 28662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa: Bootstrap starting.
I20260928 15:51:04.487488 28662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:04.488858 28662 log.cc:824] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:04.489780 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:04.477132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.22.157.254:34289)
0928 15:51:04.478270 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:04.478291 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:04.478334 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:51:04.480048 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:04.480070 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:04.480587 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:04.481080 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:04.481899 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:04.483184 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:04.484611 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:04.485477 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:04.485699 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:04.486939 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:04.486947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:04.486961 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:04.487511 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:04.487610 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:04.487663 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:04.487776 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:04.488051 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 15:51:04.488411 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":439,"threads_started":1}
I20260928 15:51:04.491935 28662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa: No bootstrap required, opened a new log
I20260928 15:51:04.493750 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:51:04.500676 28662 raft_consensus.cc:348] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40ef4ead452488981b46314057fc1aa" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34289 } }
I20260928 15:51:04.501119 28662 raft_consensus.cc:374] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:04.501195 28662 raft_consensus.cc:729] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ef4ead452488981b46314057fc1aa, State: Initialized, Role: FOLLOWER
I20260928 15:51:04.501678 28662 consensus_queue.cc:260] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40ef4ead452488981b46314057fc1aa" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34289 } }
I20260928 15:51:04.501868 28662 raft_consensus.cc:388] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:04.501973 28662 raft_consensus.cc:482] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:04.502086 28662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:04.504541 28662 raft_consensus.cc:504] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40ef4ead452488981b46314057fc1aa" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34289 } }
I20260928 15:51:04.505062 28662 leader_election.cc:302] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f40ef4ead452488981b46314057fc1aa; no voters:
I20260928 15:51:04.505626 28662 leader_election.cc:288] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:04.505803 28667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:04.506516 28667 raft_consensus.cc:686] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [term 1 LEADER]: Becoming Leader. State: Replica: f40ef4ead452488981b46314057fc1aa, State: Running, Role: LEADER
I20260928 15:51:04.507059 28667 consensus_queue.cc:237] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40ef4ead452488981b46314057fc1aa" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34289 } }
I20260928 15:51:04.507836 28662 sys_catalog.cc:564] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:04.509995 28668 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f40ef4ead452488981b46314057fc1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40ef4ead452488981b46314057fc1aa" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34289 } } }
I20260928 15:51:04.510356 28668 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:04.511593 28669 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f40ef4ead452488981b46314057fc1aa. Latest consensus state: current_term: 1 leader_uuid: "f40ef4ead452488981b46314057fc1aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40ef4ead452488981b46314057fc1aa" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 34289 } } }
I20260928 15:51:04.512006 28669 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:04.513928 28673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:04.519770 28673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:04.529199 28673 catalog_manager.cc:1347] Generated new cluster ID: 48cdc78d03c348e09ac3420951bdd810
I20260928 15:51:04.529313 28673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:04.561822 28673 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:04.563887 28673 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:04.585069 28673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f40ef4ead452488981b46314057fc1aa: Generated new TSK 0
I20260928 15:51:04.586023 28673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:51:04.611156 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:04.601834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.22.157.254:34289)
0928 15:51:04.602100 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:04.602110 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:04.602626 (+ 516us) server_negotiation.cc:407] Connection header received
0928 15:51:04.602810 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:04.602814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:04.602886 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:04.603050 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:51:04.603630 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:04.604394 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:04.606513 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:04.607051 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:04.607164 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:04.607882 (+ 718us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:51:04.610492 (+ 2610us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 15:51:04.610845 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 15:51:04.611294 28665 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 15:51:04.613723 28691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:04.601750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:34289 (local address 127.0.0.1:43466)
0928 15:51:04.602404 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:04.602427 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:04.602613 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:04.603097 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:04.603102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:04.603117 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:51:04.603441 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:04.603454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:04.604557 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:04.604562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:04.606330 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:04.606348 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:04.607270 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:04.607280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:04.607477 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:04.607543 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:51:04.610746 (+ 3203us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 15:51:04.613185 (+ 2439us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.157.254:34289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39495/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":2061,"thread_start_us":349,"threads_started":1}
I20260928 15:51:04.614800 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 28595
I20260928 15:51:04.615983 28657 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 15:51:04.623519 28692 webserver.cc:533] Webserver: sslize failed:
E20260928 15:51:04.623615 28692 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 15:51:04.712971 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 28595
2026-09-28T15:51:04Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (796 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T15:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:04Z Disabled control of system clock
I20260928 15:51:04.798993 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:33883
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:33883
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/testchainca.cert with env {}
W20260928 15:51:05.131639 28697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:05.131958 28697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:05.132139 28697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:05.139515 28697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:05.139627 28697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:05.139704 28697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:05.139827 28697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:05.139892 28697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 15:51:05.139985 28697 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 15:51:05.140074 28697 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 15:51:05.150372 28697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:33883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:33883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:05.151856 28697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:05.153330 28697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:05.166874 28702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:05.166949 28703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:05.166935 28705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:05.168023 28697 server_base.cc:1034] running on GCE node
I20260928 15:51:05.169083 28697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:05.170840 28697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:05.172106 28697 hybrid_clock.cc:648] HybridClock initialized: now 1790610665172059 us; error 38 us; skew 500 ppm
I20260928 15:51:05.172580 28697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:05.175347 28697 webserver.cc:460] Webserver started at http://127.22.157.254:34629/ using document root <none> and password file <none>
I20260928 15:51:05.176128 28697 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:05.176251 28697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:05.176592 28697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:05.179760 28697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "3e38bb0f482c4995ad0248f30609f754"
format_stamp: "Formatted at 2026-09-28 15:51:05 on dist-test-slave-1vw0"
server_key: "48560dccb3d9b6209c3c7a4f60534a23"
server_key_iv: "8253d6edf22b5a765c42c2b76d5dceed"
server_key_version: "encryptionkey@0"
I20260928 15:51:05.180614 28697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "3e38bb0f482c4995ad0248f30609f754"
format_stamp: "Formatted at 2026-09-28 15:51:05 on dist-test-slave-1vw0"
server_key: "48560dccb3d9b6209c3c7a4f60534a23"
server_key_iv: "8253d6edf22b5a765c42c2b76d5dceed"
server_key_version: "encryptionkey@0"
I20260928 15:51:05.185938 28697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:51:05.189798 28711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:05.191429 28697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:05.191669 28697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "3e38bb0f482c4995ad0248f30609f754"
format_stamp: "Formatted at 2026-09-28 15:51:05 on dist-test-slave-1vw0"
server_key: "48560dccb3d9b6209c3c7a4f60534a23"
server_key_iv: "8253d6edf22b5a765c42c2b76d5dceed"
server_key_version: "encryptionkey@0"
I20260928 15:51:05.191916 28697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:05.215099 28697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:05.216295 28697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:05.216696 28697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:05.237406 28697 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:33883
I20260928 15:51:05.237421 28762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:33883 every 8 connection(s)
I20260928 15:51:05.239125 28697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:05.240515 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 28697
I20260928 15:51:05.240772 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:05.241436 23159 external_mini_cluster.cc:1428] Setting key 627c27e699f39c0ab61650654a796009
I20260928 15:51:05.244664 28763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:05.255859 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:05.243293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.22.157.254:33883)
0928 15:51:05.244378 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:05.244390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:05.245011 (+ 621us) server_negotiation.cc:407] Connection header received
0928 15:51:05.246816 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:05.246839 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:05.247339 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:05.247805 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:05.248409 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:05.249845 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:05.251253 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:05.251953 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:05.252162 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:05.253486 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:05.253493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:05.253519 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:05.254125 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:05.254209 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:05.254233 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:05.254366 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:05.254602 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 15:51:05.254922 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":415,"threads_started":1}
I20260928 15:51:05.255894 28763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754: Bootstrap starting.
I20260928 15:51:05.259302 28763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:05.259791 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:51:05.260462 28763 log.cc:824] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:05.263710 28763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754: No bootstrap required, opened a new log
I20260928 15:51:05.271553 28763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38bb0f482c4995ad0248f30609f754" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33883 } }
I20260928 15:51:05.271975 28763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:05.272102 28763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e38bb0f482c4995ad0248f30609f754, State: Initialized, Role: FOLLOWER
I20260928 15:51:05.272771 28763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38bb0f482c4995ad0248f30609f754" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33883 } }
I20260928 15:51:05.273041 28763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:05.273197 28763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:05.273383 28763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:05.275944 28763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38bb0f482c4995ad0248f30609f754" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33883 } }
I20260928 15:51:05.276499 28763 leader_election.cc:302] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e38bb0f482c4995ad0248f30609f754; no voters:
I20260928 15:51:05.277179 28763 leader_election.cc:288] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:05.277333 28768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:05.278035 28768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [term 1 LEADER]: Becoming Leader. State: Replica: 3e38bb0f482c4995ad0248f30609f754, State: Running, Role: LEADER
I20260928 15:51:05.278595 28768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38bb0f482c4995ad0248f30609f754" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33883 } }
I20260928 15:51:05.279294 28763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:05.281813 28770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e38bb0f482c4995ad0248f30609f754. Latest consensus state: current_term: 1 leader_uuid: "3e38bb0f482c4995ad0248f30609f754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38bb0f482c4995ad0248f30609f754" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33883 } } }
I20260928 15:51:05.282238 28770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:05.282445 28769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e38bb0f482c4995ad0248f30609f754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38bb0f482c4995ad0248f30609f754" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33883 } } }
I20260928 15:51:05.282768 28769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:05.284509 28777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:05.289985 28777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:05.298942 28777 catalog_manager.cc:1347] Generated new cluster ID: 11180f316a0a4ea8be0ce07473c36307
I20260928 15:51:05.299058 28777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:05.315330 28777 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:05.317268 28777 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:05.328575 28777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e38bb0f482c4995ad0248f30609f754: Generated new TSK 0
I20260928 15:51:05.329733 28777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:51:05.419859 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:05.349758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.22.157.254:33883)
0928 15:51:05.349996 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:05.350003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:05.350359 (+ 356us) server_negotiation.cc:407] Connection header received
0928 15:51:05.350584 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:05.350588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:05.350662 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:05.350802 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0928 15:51:05.351347 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:05.352103 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:05.354319 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:05.354796 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:05.354889 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:05.355516 (+ 627us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 15:51:05.419194 (+ 63678us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 15:51:05.419623 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45322: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 15:51:05.420032 28766 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45322: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 15:51:05.420297 28793 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:05.349578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:33883 (local address 127.0.0.1:45322)
0928 15:51:05.350240 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:05.350262 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:05.350419 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:05.350846 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:05.350853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:05.350866 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 15:51:05.351190 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:05.351204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:05.352257 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:05.352263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:05.354172 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:05.354187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:05.354973 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:05.354980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:05.355173 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:05.355221 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 15:51:05.419529 (+ 64308us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 15:51:05.419863 (+ 334us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.157.254:33883: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":325,"threads_started":1}
I20260928 15:51:05.421450 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 28697
I20260928 15:51:05.422713 28758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:05.524024 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 28697
2026-09-28T15:51:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (790 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (428 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (440 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:51:06 dist-test-slave-1vw0 krb5kdc[28808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:51:06 dist-test-slave-1vw0 krb5kdc[28808](info): set up 2 sockets
Sep 28 15:51:06 dist-test-slave-1vw0 krb5kdc[28808](info): commencing operation
krb5kdc: starting...
W20260928 15:51:07.498549 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:51:07 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:51:07.745879 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42929
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:42929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:08.093071 28821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:08.093452 28821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:08.093626 28821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:08.101207 28821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:08.101389 28821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:08.101507 28821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:08.101621 28821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:08.101698 28821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:08.111954 28821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:42929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:08.113454 28821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:08.114847 28821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:08.127650 28826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:08.127622 28829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:08.127856 28827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:08.129447 28821 server_base.cc:1034] running on GCE node
I20260928 15:51:08.130342 28821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:08.131937 28821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:08.133198 28821 hybrid_clock.cc:648] HybridClock initialized: now 1790610668133156 us; error 56 us; skew 500 ppm
I20260928 15:51:08.133594 28821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:08 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610668, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:08.137161 28821 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:51:08.139060 28821 webserver.cc:460] Webserver started at http://127.22.157.254:45623/ using document root <none> and password file <none>
I20260928 15:51:08.139889 28821 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:08.140125 28821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:08.140522 28821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:08.143810 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "b09985c06b2f4b9ab69a87c2b8246d2a"
format_stamp: "Formatted at 2026-09-28 15:51:08 on dist-test-slave-1vw0"
server_key: "3e325c5dba954cc1d1af906deacaeb51"
server_key_iv: "979b4648d5eaa1414b456085c4bc7e3f"
server_key_version: "encryptionkey@0"
I20260928 15:51:08.144663 28821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "b09985c06b2f4b9ab69a87c2b8246d2a"
format_stamp: "Formatted at 2026-09-28 15:51:08 on dist-test-slave-1vw0"
server_key: "3e325c5dba954cc1d1af906deacaeb51"
server_key_iv: "979b4648d5eaa1414b456085c4bc7e3f"
server_key_version: "encryptionkey@0"
I20260928 15:51:08.150336 28821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:51:08.154090 28836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:08.155761 28821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:08.156026 28821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "b09985c06b2f4b9ab69a87c2b8246d2a"
format_stamp: "Formatted at 2026-09-28 15:51:08 on dist-test-slave-1vw0"
server_key: "3e325c5dba954cc1d1af906deacaeb51"
server_key_iv: "979b4648d5eaa1414b456085c4bc7e3f"
server_key_version: "encryptionkey@0"
I20260928 15:51:08.156286 28821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:08.182040 28821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:08.186868 28821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:08.187217 28821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:08.207099 28821 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:42929
I20260928 15:51:08.207142 28887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:42929 every 8 connection(s)
I20260928 15:51:08.208564 28821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:08.209699 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 28821
I20260928 15:51:08.209961 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:08.210873 23159 external_mini_cluster.cc:1428] Setting key 1418767790bf66ebfb85ba47c0e0c17b
I20260928 15:51:08.214135 28888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:51:08 dist-test-slave-1vw0 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:51:08.226464 28888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a: Bootstrap starting.
I20260928 15:51:08.230304 28888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:08.231534 28888 log.cc:824] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:08.234534 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:08.213307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.22.157.254:42929)
0928 15:51:08.214171 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:08.214185 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:08.214229 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:51:08.216014 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:08.216039 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:08.216450 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:08.216933 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:08.218485 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:08.219905 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:08.221333 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:08.222083 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:08.222304 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:08.226495 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:08.226505 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:08.226524 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:08.226571 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:08.229860 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:08.230763 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:08.230770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:08.230780 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:08.230967 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:08.231392 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:08.231397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:08.231399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:08.232040 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:08.232421 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:08.232983 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 15:51:08.233475 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":356,"threads_started":1}
I20260928 15:51:08.235172 28888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a: No bootstrap required, opened a new log
I20260928 15:51:08.243976 28888 raft_consensus.cc:348] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09985c06b2f4b9ab69a87c2b8246d2a" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42929 } }
I20260928 15:51:08.244477 28888 raft_consensus.cc:374] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:08.244609 28888 raft_consensus.cc:729] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09985c06b2f4b9ab69a87c2b8246d2a, State: Initialized, Role: FOLLOWER
I20260928 15:51:08.245252 28888 consensus_queue.cc:260] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09985c06b2f4b9ab69a87c2b8246d2a" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42929 } }
I20260928 15:51:08.245515 28888 raft_consensus.cc:388] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:08.245680 28888 raft_consensus.cc:482] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:08.245858 28888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:08.248127 28888 raft_consensus.cc:504] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09985c06b2f4b9ab69a87c2b8246d2a" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42929 } }
I20260928 15:51:08.248715 28888 leader_election.cc:302] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b09985c06b2f4b9ab69a87c2b8246d2a; no voters:
I20260928 15:51:08.249609 28888 leader_election.cc:288] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:08.249771 28893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:08.250569 28893 raft_consensus.cc:686] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [term 1 LEADER]: Becoming Leader. State: Replica: b09985c06b2f4b9ab69a87c2b8246d2a, State: Running, Role: LEADER
I20260928 15:51:08.251204 28893 consensus_queue.cc:237] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09985c06b2f4b9ab69a87c2b8246d2a" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42929 } }
I20260928 15:51:08.252012 28888 sys_catalog.cc:564] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:08.254149 28895 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b09985c06b2f4b9ab69a87c2b8246d2a. Latest consensus state: current_term: 1 leader_uuid: "b09985c06b2f4b9ab69a87c2b8246d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09985c06b2f4b9ab69a87c2b8246d2a" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42929 } } }
I20260928 15:51:08.255048 28895 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:08.256392 28894 sys_catalog.cc:455] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b09985c06b2f4b9ab69a87c2b8246d2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b09985c06b2f4b9ab69a87c2b8246d2a" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42929 } } }
I20260928 15:51:08.256909 28894 sys_catalog.cc:458] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:08.259904 28902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:08.265903 28902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:08.277588 28902 catalog_manager.cc:1347] Generated new cluster ID: 257c586e7812498da2b66ebd3e742c59
I20260928 15:51:08.277731 28902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:08.293677 28902 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:08.295876 28902 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:08.309176 28902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b09985c06b2f4b9ab69a87c2b8246d2a: Generated new TSK 0
I20260928 15:51:08.310520 28902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:51:08.373191 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42929
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:08.741394 28914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:08.741858 28914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:08.742058 28914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:08.752177 28914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:08.752380 28914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:08.752535 28914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:08.763661 28914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:08.765133 28914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:08.766538 28914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:08.781739 28919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:08.781662 28920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:08.781699 28922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:08.782058 28914 server_base.cc:1034] running on GCE node
I20260928 15:51:08.783174 28914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:08.784338 28914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:08.785642 28914 hybrid_clock.cc:648] HybridClock initialized: now 1790610668785582 us; error 99 us; skew 500 ppm
I20260928 15:51:08.786103 28914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:08 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610668, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:08.790052 28914 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:51:08.791882 28914 webserver.cc:460] Webserver started at http://127.22.157.193:44319/ using document root <none> and password file <none>
I20260928 15:51:08.792688 28914 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:08.792927 28914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:08.793283 28914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:08.796692 28914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "0996ddcd96c24b5d8eecdb4fc1ba849e"
format_stamp: "Formatted at 2026-09-28 15:51:08 on dist-test-slave-1vw0"
server_key: "f128927c468a026f2fe4e908445aaf59"
server_key_iv: "ddd6ff0453f457b5d14721c26f61b5e8"
server_key_version: "encryptionkey@0"
I20260928 15:51:08.797729 28914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "0996ddcd96c24b5d8eecdb4fc1ba849e"
format_stamp: "Formatted at 2026-09-28 15:51:08 on dist-test-slave-1vw0"
server_key: "f128927c468a026f2fe4e908445aaf59"
server_key_iv: "ddd6ff0453f457b5d14721c26f61b5e8"
server_key_version: "encryptionkey@0"
I20260928 15:51:08.804030 28914 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:51:08.808429 28930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:08.810410 28914 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:51:08.810657 28914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "0996ddcd96c24b5d8eecdb4fc1ba849e"
format_stamp: "Formatted at 2026-09-28 15:51:08 on dist-test-slave-1vw0"
server_key: "f128927c468a026f2fe4e908445aaf59"
server_key_iv: "ddd6ff0453f457b5d14721c26f61b5e8"
server_key_version: "encryptionkey@0"
I20260928 15:51:08.810894 28914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:08.830359 28914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:08.835788 28914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:08.836171 28914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:08.837481 28914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:08.839066 28914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:08.839144 28914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:08.839277 28914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:08.839344 28914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:08.876109 28914 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:45313
I20260928 15:51:08.876170 29042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:45313 every 8 connection(s)
I20260928 15:51:08.878156 28914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:08.888322 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 28914
I20260928 15:51:08.888558 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:51:08.889348 23159 external_mini_cluster.cc:1428] Setting key db02b8566ca0284505cec3226e708573
Sep 28 15:51:08 dist-test-slave-1vw0 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610668, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:51:08.900837 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:08.881537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:57671 (local address 127.22.157.254:42929)
0928 15:51:08.882070 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:08.882074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:08.883232 (+ 1158us) server_negotiation.cc:407] Connection header received
0928 15:51:08.885214 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:08.885220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:08.885322 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:08.885447 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:08.887089 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:08.887839 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:08.889737 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:08.890220 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:08.890354 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:08.894639 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:08.894644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:08.894647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:08.894683 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:08.897392 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:08.898241 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:08.898246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:08.898248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:08.898374 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:08.898921 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:08.898925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:08.898928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:08.899227 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:08.899352 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:08.900325 (+ 973us) server_negotiation.cc:299] Negotiation successful
0928 15:51:08.900518 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":305,"threads_started":1}
I20260928 15:51:08.902566 29046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:08.882117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:42929 (local address 127.22.157.193:57671)
0928 15:51:08.883050 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:08.883100 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:08.884906 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:08.885716 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:08.885725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:08.886289 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:08.886895 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:08.886913 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:08.888025 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:08.888029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:08.889567 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:08.889585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:08.890574 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:08.890584 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:08.890921 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:08.891847 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:08.891872 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:08.894355 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:08.897576 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:08.897586 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:08.897605 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:08.898091 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:08.898502 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:08.898507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:08.898511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:08.898727 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:08.899364 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:08.899376 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:08.899884 (+ 508us) client_negotiation.cc:769] Sending connection context
0928 15:51:08.900210 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 15:51:08.900866 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":322,"threads_started":1}
I20260928 15:51:08.904330 29043 heartbeater.cc:347] Connected to a master server at 127.22.157.254:42929
I20260928 15:51:08.904704 29043 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:08.905691 29043 heartbeater.cc:511] Master 127.22.157.254:42929 requested a full tablet report, sending...
I20260928 15:51:08.908283 28853 ts_manager.cc:194] Registered new tserver with Master: 0996ddcd96c24b5d8eecdb4fc1ba849e (127.22.157.193:45313)
I20260928 15:51:08.909833 28853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:57671
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:51:08.941454 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42929
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:09.283298 29049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:09.283612 29049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:09.283774 29049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:09.291157 29049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:09.291280 29049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:09.291397 29049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:51:09.301445 29049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:09.302816 29049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:09.304214 29049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:09.318328 29054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:09.318279 29055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:09.318248 29057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:09.319744 29049 server_base.cc:1034] running on GCE node
I20260928 15:51:09.320429 29049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:09.321547 29049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:09.322764 29049 hybrid_clock.cc:648] HybridClock initialized: now 1790610669322716 us; error 46 us; skew 500 ppm
I20260928 15:51:09.323113 29049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:09 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:09.326642 29049 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:51:09.328297 29049 webserver.cc:460] Webserver started at http://127.22.157.194:37039/ using document root <none> and password file <none>
I20260928 15:51:09.329134 29049 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:09.329282 29049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:09.329612 29049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:09.332691 29049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "e7d012912d2143feb9ac983b90d44bf8"
format_stamp: "Formatted at 2026-09-28 15:51:09 on dist-test-slave-1vw0"
server_key: "9c7153155bfb83a24e0ba29ec6db54e2"
server_key_iv: "c61f7bbc53520978abffddfe2541dee5"
server_key_version: "encryptionkey@0"
I20260928 15:51:09.333714 29049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "e7d012912d2143feb9ac983b90d44bf8"
format_stamp: "Formatted at 2026-09-28 15:51:09 on dist-test-slave-1vw0"
server_key: "9c7153155bfb83a24e0ba29ec6db54e2"
server_key_iv: "c61f7bbc53520978abffddfe2541dee5"
server_key_version: "encryptionkey@0"
I20260928 15:51:09.339198 29049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:51:09.342913 29064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:09.344455 29049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:51:09.344686 29049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "e7d012912d2143feb9ac983b90d44bf8"
format_stamp: "Formatted at 2026-09-28 15:51:09 on dist-test-slave-1vw0"
server_key: "9c7153155bfb83a24e0ba29ec6db54e2"
server_key_iv: "c61f7bbc53520978abffddfe2541dee5"
server_key_version: "encryptionkey@0"
I20260928 15:51:09.345002 29049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:09.367445 29049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:09.372184 29049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:09.372506 29049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:09.373626 29049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:09.375165 29049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:09.375276 29049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:09.375429 29049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:09.375504 29049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:09.409142 29049 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:33751
I20260928 15:51:09.409188 29178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:33751 every 8 connection(s)
I20260928 15:51:09.410533 29049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:51:09.415395 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 29049
I20260928 15:51:09.415581 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:51:09.416208 23159 external_mini_cluster.cc:1428] Setting key b65b793f71d1a988642188b4ecf17ec8
Sep 28 15:51:09 dist-test-slave-1vw0 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
I20260928 15:51:09.441045 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:09.413375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:45431 (local address 127.22.157.254:42929)
0928 15:51:09.414010 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:09.414015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:09.415157 (+ 1142us) server_negotiation.cc:407] Connection header received
0928 15:51:09.417746 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:09.417750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:09.417817 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:09.417914 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:09.420048 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:09.420944 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:09.423565 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:09.424030 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:09.424135 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:09.432772 (+ 8637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:09.432783 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:09.432791 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:09.432850 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:09.437116 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:09.438168 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:09.438176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:09.438179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:09.438374 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:09.438942 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:09.438948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:09.438951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:09.439392 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:09.439594 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:09.440447 (+ 853us) server_negotiation.cc:299] Negotiation successful
0928 15:51:09.440644 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":400,"threads_started":1}
I20260928 15:51:09.442459 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:09.414061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:42929 (local address 127.22.157.194:45431)
0928 15:51:09.414882 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:09.415005 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:09.417430 (+ 2425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:09.418231 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:09.418241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:09.418976 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:09.419832 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:09.419862 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:09.421115 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:09.421123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:09.423376 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:09.423408 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:09.424291 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:09.424301 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:09.424653 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:09.429869 (+ 5216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:09.429905 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:09.432369 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:09.437376 (+ 5007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:09.437386 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:09.437403 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:09.437954 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:09.438565 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:09.438570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:09.438572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:09.438797 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:09.439562 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:09.439569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:09.439947 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 15:51:09.440306 (+ 359us) client_negotiation.cc:240] Negotiation successful
0928 15:51:09.440833 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":354,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
I20260928 15:51:09.444628 29179 heartbeater.cc:347] Connected to a master server at 127.22.157.254:42929
I20260928 15:51:09.445030 29179 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:09.445865 29179 heartbeater.cc:511] Master 127.22.157.254:42929 requested a full tablet report, sending...
I20260928 15:51:09.447610 28853 ts_manager.cc:194] Registered new tserver with Master: e7d012912d2143feb9ac983b90d44bf8 (127.22.157.194:33751)
I20260928 15:51:09.448511 28853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:45431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:51:09.472513 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42929
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:09.810514 29185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:09.810832 29185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:09.810930 29185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:09.818852 29185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:09.818971 29185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:09.819090 29185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:51:09.829351 29185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:09.830686 29185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:09.831971 29185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:09.846015 29193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:09.845961 29191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:09.845920 29190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:09.847119 29185 server_base.cc:1034] running on GCE node
I20260928 15:51:09.847851 29185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:09.848989 29185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:09.850219 29185 hybrid_clock.cc:648] HybridClock initialized: now 1790610669850184 us; error 38 us; skew 500 ppm
I20260928 15:51:09.850531 29185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:09 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:09.854329 29185 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:51:09.856076 29185 webserver.cc:460] Webserver started at http://127.22.157.195:36261/ using document root <none> and password file <none>
I20260928 15:51:09.856993 29185 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:09.857139 29185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:09.857496 29185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:09.860702 29185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "7a25d3ce53cf4a9387fcc9ce64be3606"
format_stamp: "Formatted at 2026-09-28 15:51:09 on dist-test-slave-1vw0"
server_key: "e2b1874d430d101811345d57319e6d3f"
server_key_iv: "87e872f73e610063a0c16a68488b8d3f"
server_key_version: "encryptionkey@0"
I20260928 15:51:09.861620 29185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "7a25d3ce53cf4a9387fcc9ce64be3606"
format_stamp: "Formatted at 2026-09-28 15:51:09 on dist-test-slave-1vw0"
server_key: "e2b1874d430d101811345d57319e6d3f"
server_key_iv: "87e872f73e610063a0c16a68488b8d3f"
server_key_version: "encryptionkey@0"
I20260928 15:51:09.867066 29185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:51:09.871368 29200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:09.872958 29185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:09.873142 29185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "7a25d3ce53cf4a9387fcc9ce64be3606"
format_stamp: "Formatted at 2026-09-28 15:51:09 on dist-test-slave-1vw0"
server_key: "e2b1874d430d101811345d57319e6d3f"
server_key_iv: "87e872f73e610063a0c16a68488b8d3f"
server_key_version: "encryptionkey@0"
I20260928 15:51:09.873356 29185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:09.911311 29185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:09.913569 29043 heartbeater.cc:503] Master 127.22.157.254:42929 was elected leader, sending a full tablet report...
I20260928 15:51:09.916474 29185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:09.916817 29185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:09.917888 29185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:09.919278 29185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:09.919353 29185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:09.919448 29185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:09.919502 29185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:09.954231 29185 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:46609
I20260928 15:51:09.954291 29314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:46609 every 8 connection(s)
I20260928 15:51:09.955629 29185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:51:09.956208 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 29185
I20260928 15:51:09.956398 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:51:09.957082 23159 external_mini_cluster.cc:1428] Setting key c89bad6769273a323b1e777d1bb44715
Sep 28 15:51:09 dist-test-slave-1vw0 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610669, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:51:09.977730 29317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:09.958555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:38777 (local address 127.22.157.254:42929)
0928 15:51:09.959045 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:09.959051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:09.960228 (+ 1177us) server_negotiation.cc:407] Connection header received
0928 15:51:09.962544 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:09.962549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:09.962657 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:09.962774 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:09.964327 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:09.965217 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:09.966823 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:09.967230 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:09.967372 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:09.971310 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:09.971316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:09.971319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:09.971356 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:09.974542 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:09.975280 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:09.975284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:09.975286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:09.975463 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:09.975866 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:09.975869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:09.975871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:09.976141 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:09.976260 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:09.977186 (+ 926us) server_negotiation.cc:299] Negotiation successful
0928 15:51:09.977412 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":305,"threads_started":1}
I20260928 15:51:09.978930 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:09.959174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:42929 (local address 127.22.157.195:38777)
0928 15:51:09.960084 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:09.960125 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:09.962277 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:09.963038 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:09.963046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:09.963576 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:09.964156 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:09.964174 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:09.965376 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:09.965381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:09.966662 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:09.966674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:09.967473 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:09.967480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:09.967744 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:09.968418 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:09.968446 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:09.970968 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:09.974696 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:09.974704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:09.974719 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:09.975172 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:09.975565 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:09.975569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:09.975571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:09.975769 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:09.976244 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:09.976250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:09.976692 (+ 442us) client_negotiation.cc:769] Sending connection context
0928 15:51:09.977040 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 15:51:09.977620 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":391,"threads_started":1}
I20260928 15:51:09.980381 29315 heartbeater.cc:347] Connected to a master server at 127.22.157.254:42929
I20260928 15:51:09.980667 29315 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:09.981431 29315 heartbeater.cc:511] Master 127.22.157.254:42929 requested a full tablet report, sending...
I20260928 15:51:09.982810 28853 ts_manager.cc:194] Registered new tserver with Master: 7a25d3ce53cf4a9387fcc9ce64be3606 (127.22.157.195:46609)
I20260928 15:51:09.983698 28853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:38777
I20260928 15:51:09.986091 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/test-admin.keytab.
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:10.015560 23159 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 15:51:10.021202 29323 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:10.018573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.22.157.193:45313)
0928 15:51:10.019232 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:10.019239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:10.019264 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:51:10.019322 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:10.019329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:10.019567 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:10.019741 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:10.020700 (+ 959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57872: BlockingRecv error: recv got EOF from 127.0.0.1:57872 (error 108)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":331,"threads_started":1}
W20260928 15:51:10.021332 23159 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5cc' of type FILE
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:10.023087 23159 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:10.024523 23159 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:51:10.041875 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:10.026359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.22.157.193:45313)
0928 15:51:10.026660 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:10.026666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:10.026879 (+ 213us) server_negotiation.cc:407] Connection header received
0928 15:51:10.027034 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:10.027038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:10.027103 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:10.027189 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:10.028220 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.029363 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.030713 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.031326 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.031440 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:10.035391 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:10.035406 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:10.035430 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:10.035473 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:10.038595 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.039557 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.039566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.039569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.039716 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.040126 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.040130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.040132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.040570 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:10.040882 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:10.041275 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 15:51:10.041611 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 15:51:10.048265 29323 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:10.046362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.22.157.193:45313)
0928 15:51:10.046529 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:10.046534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:10.046923 (+ 389us) server_negotiation.cc:407] Connection header received
0928 15:51:10.047099 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:10.047102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:10.047176 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:10.047257 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:10.047979 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57886: BlockingRecv error: recv got EOF from 127.0.0.1:57886 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 15:51:10.048938 23159 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5cc)
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:10.050601 23159 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:10.052147 23159 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:51:10.068437 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:10.054123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.22.157.193:45313)
0928 15:51:10.054382 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:10.054389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:10.054578 (+ 189us) server_negotiation.cc:407] Connection header received
0928 15:51:10.054802 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:10.054808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:10.054875 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:10.054977 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:10.056180 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.057116 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.058405 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.058871 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.059022 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:10.063000 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:10.063005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:10.063009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:10.063054 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:10.065772 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.066495 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.066500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.066502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.066634 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.067027 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.067040 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.067042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.067324 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:10.067598 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:10.067923 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 15:51:10.068150 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260928 15:51:10.072945 29323 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:10.071145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.22.157.193:45313)
0928 15:51:10.071306 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:10.071309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:10.071687 (+ 378us) server_negotiation.cc:407] Connection header received
0928 15:51:10.071808 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:10.071812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:10.071882 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:10.071960 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:10.072621 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57914: BlockingRecv error: recv got EOF from 127.0.0.1:57914 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 15:51:10.073137 23159 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790610618730370-23159-0/krb5kdc/krb5cc)
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:10.074715 23159 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:10.076040 23159 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:51:10.091558 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:10.077795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.22.157.193:45313)
0928 15:51:10.078057 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:10.078066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:10.078361 (+ 295us) server_negotiation.cc:407] Connection header received
0928 15:51:10.078562 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:10.078568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:10.078643 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:10.078747 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:10.079761 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.080538 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.081911 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.082360 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.082503 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:10.086263 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:10.086268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:10.086272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:10.086307 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:10.088877 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.089545 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.089553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.089559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.089722 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.090112 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.090123 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.090125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.090412 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:10.090741 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:10.091010 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 15:51:10.091239 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 15:51:10.093268 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 28914
I20260928 15:51:10.105575 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:10.093940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.22.157.193:45313)
0928 15:51:10.094101 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:10.094114 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:10.094479 (+ 365us) server_negotiation.cc:407] Connection header received
0928 15:51:10.094651 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:10.094655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:10.094709 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:10.094785 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:10.095843 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.096526 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.097811 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.098234 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.098327 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:10.100612 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:10.100616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:10.100619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:10.100652 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:10.103172 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.103855 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.103858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.103861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.103996 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.104344 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.104347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.104349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.104601 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:10.104879 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:10.105150 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 15:51:10.105316 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:51:10.106148 29038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:10.191437 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 28914
I20260928 15:51:10.211184 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 29049
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:51:10.228102 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:10.212088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.22.157.194:33751)
0928 15:51:10.212654 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:10.212661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:10.212687 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:51:10.212810 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:10.212824 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:10.213061 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:10.213237 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:10.214362 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.215379 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.216678 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.217421 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.217571 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:10.221622 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:10.221636 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:10.221651 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:10.221708 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:10.224716 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.225641 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.225649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.225652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.225809 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.226263 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.226268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.226272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.226663 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:10.226994 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:10.227314 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 15:51:10.227594 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":278,"threads_started":1}
I20260928 15:51:10.229595 29174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:10.313642 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 29049
I20260928 15:51:10.330088 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 29185
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[28808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:51:10.347280 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:10.331163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.22.157.195:46609)
0928 15:51:10.332080 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:10.332088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:10.332129 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:51:10.332197 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:10.332204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:10.332502 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:10.332666 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:10.333999 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.335102 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.336485 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:10.337216 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:10.337323 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:10.341394 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:10.341407 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:10.341424 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:10.341516 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:10.344529 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.345408 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.345415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.345417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.345602 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:10.345939 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:10.345942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:10.345944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:10.346301 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:10.346587 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:10.346780 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 15:51:10.346990 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":568,"threads_started":1}
I20260928 15:51:10.348899 29310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:10.434311 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 29185
I20260928 15:51:10.451042 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 28821
I20260928 15:51:10.452456 28883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:10.558192 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 28821
2026-09-28T15:51:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4171 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[29343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[29343](info): set up 2 sockets
Sep 28 15:51:10 dist-test-slave-1vw0 krb5kdc[29343](info): commencing operation
krb5kdc: starting...
W20260928 15:51:11.674122 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:51:11 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.254.keytab.
Entry for principal oryx/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.254.keytab.
I20260928 15:51:11.917413 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:46257
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.254.keytab
--principal=oryx/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:46257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:12.260407 29356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:12.260794 29356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:12.260979 29356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:12.268551 29356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:12.268697 29356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:12.268821 29356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:12.268930 29356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:12.268996 29356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:12.279699 29356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:46257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.254.keytab
--principal=oryx/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:12.281294 29356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:12.282718 29356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:12.296037 29361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:12.295946 29364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:12.296005 29362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:12.296428 29356 server_base.cc:1034] running on GCE node
I20260928 15:51:12.297593 29356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:12.299131 29356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:12.300390 29356 hybrid_clock.cc:648] HybridClock initialized: now 1790610672300331 us; error 49 us; skew 500 ppm
I20260928 15:51:12.300854 29356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:12 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610672, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:12.304559 29356 init.cc:377] Logged in from keytab as oryx/127.22.157.254@KRBTEST.COM (short username oryx)
I20260928 15:51:12.306332 29356 webserver.cc:460] Webserver started at http://127.22.157.254:41425/ using document root <none> and password file <none>
I20260928 15:51:12.307147 29356 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:12.307268 29356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:12.307545 29356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:12.311040 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "4d59f65974d44666b7524a34baf610b7"
format_stamp: "Formatted at 2026-09-28 15:51:12 on dist-test-slave-1vw0"
server_key: "91fad165f753bdc4f809a8bb3053b80a"
server_key_iv: "0a3883d540d1dc3b9e008f9ca99c5816"
server_key_version: "encryptionkey@0"
I20260928 15:51:12.311864 29356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "4d59f65974d44666b7524a34baf610b7"
format_stamp: "Formatted at 2026-09-28 15:51:12 on dist-test-slave-1vw0"
server_key: "91fad165f753bdc4f809a8bb3053b80a"
server_key_iv: "0a3883d540d1dc3b9e008f9ca99c5816"
server_key_version: "encryptionkey@0"
I20260928 15:51:12.318321 29356 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 15:51:12.322186 29371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:12.323676 29356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:12.323880 29356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "4d59f65974d44666b7524a34baf610b7"
format_stamp: "Formatted at 2026-09-28 15:51:12 on dist-test-slave-1vw0"
server_key: "91fad165f753bdc4f809a8bb3053b80a"
server_key_iv: "0a3883d540d1dc3b9e008f9ca99c5816"
server_key_version: "encryptionkey@0"
I20260928 15:51:12.324137 29356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:12.342465 29356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:12.346743 29356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:12.347071 29356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:12.366644 29356 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:46257
I20260928 15:51:12.366674 29423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:46257 every 8 connection(s)
I20260928 15:51:12.368161 29356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:12.371400 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 29356
I20260928 15:51:12.371611 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:12.372457 23159 external_mini_cluster.cc:1428] Setting key bbd0fb4fdd7997eed22382911a799220
I20260928 15:51:12.372615 29424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:12.387782 29424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: Bootstrap starting.
Sep 28 15:51:12 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:12.391426 29424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:12.392513 29424 log.cc:824] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:12.395541 29424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: No bootstrap required, opened a new log
I20260928 15:51:12.396662 29427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:12.374667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51934 (local address 127.22.157.254:46257)
0928 15:51:12.375503 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:12.375515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:12.375559 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:51:12.377494 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:12.377516 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:12.377920 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:12.378272 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:12.380161 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:12.381604 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:12.383395 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:12.384136 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:12.384293 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:12.389078 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:12.389086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:12.389102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:12.389144 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:12.392495 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:12.393271 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:12.393279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:12.393288 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:12.393469 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:12.393862 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:12.393868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:12.393871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:12.394417 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:12.394758 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:12.395165 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 15:51:12.395645 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":378,"threads_started":1}
I20260928 15:51:12.403613 29424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } }
I20260928 15:51:12.404035 29424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:12.404183 29424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d59f65974d44666b7524a34baf610b7, State: Initialized, Role: FOLLOWER
I20260928 15:51:12.404899 29424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } }
I20260928 15:51:12.405109 29424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:12.405211 29424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:12.405339 29424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:12.408133 29424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } }
I20260928 15:51:12.408757 29424 leader_election.cc:302] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d59f65974d44666b7524a34baf610b7; no voters:
I20260928 15:51:12.409343 29424 leader_election.cc:288] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:12.409512 29429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:12.410207 29429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 1 LEADER]: Becoming Leader. State: Replica: 4d59f65974d44666b7524a34baf610b7, State: Running, Role: LEADER
I20260928 15:51:12.410760 29429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } }
I20260928 15:51:12.411522 29424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:12.413805 29431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d59f65974d44666b7524a34baf610b7. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } }
I20260928 15:51:12.414196 29431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:12.416092 29430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } }
I20260928 15:51:12.416427 29430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:12.416566 29435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:12.422578 29435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:12.432947 29435 catalog_manager.cc:1347] Generated new cluster ID: 5b1dfd5385a4479081dbef524155c4f6
I20260928 15:51:12.433082 29435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:12.459209 29435 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:12.461643 29435 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:12.475852 29435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: Generated new TSK 0
I20260928 15:51:12.477108 29435 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab.
Entry for principal oryx/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab.
I20260928 15:51:12.544857 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46257
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab
--principal=oryx/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:12.890076 29450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:12.890437 29450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:12.890679 29450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:12.899722 29450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:12.899921 29450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:12.900135 29450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:12.910662 29450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab
--principal=oryx/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:12.912182 29450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:12.913812 29450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:12.928576 29456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:12.928678 29458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:12.928586 29455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:12.929531 29450 server_base.cc:1034] running on GCE node
I20260928 15:51:12.930202 29450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:12.931233 29450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:12.932497 29450 hybrid_clock.cc:648] HybridClock initialized: now 1790610672932428 us; error 55 us; skew 500 ppm
I20260928 15:51:12.932931 29450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:12 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610672, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:12.936409 29450 init.cc:377] Logged in from keytab as oryx/127.22.157.193@KRBTEST.COM (short username oryx)
I20260928 15:51:12.938087 29450 webserver.cc:460] Webserver started at http://127.22.157.193:34345/ using document root <none> and password file <none>
I20260928 15:51:12.938778 29450 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:12.938884 29450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:12.939154 29450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:12.942204 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "d09dac1b78ef4becac2379eaf9bef297"
format_stamp: "Formatted at 2026-09-28 15:51:12 on dist-test-slave-1vw0"
server_key: "0d484104299c2f1c4f65836daf2361a0"
server_key_iv: "e8bbdd2f7549c8820d61844dae6fc3ef"
server_key_version: "encryptionkey@0"
I20260928 15:51:12.943004 29450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "d09dac1b78ef4becac2379eaf9bef297"
format_stamp: "Formatted at 2026-09-28 15:51:12 on dist-test-slave-1vw0"
server_key: "0d484104299c2f1c4f65836daf2361a0"
server_key_iv: "e8bbdd2f7549c8820d61844dae6fc3ef"
server_key_version: "encryptionkey@0"
I20260928 15:51:12.948316 29450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:51:12.951900 29465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:12.953482 29450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:12.953672 29450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "d09dac1b78ef4becac2379eaf9bef297"
format_stamp: "Formatted at 2026-09-28 15:51:12 on dist-test-slave-1vw0"
server_key: "0d484104299c2f1c4f65836daf2361a0"
server_key_iv: "e8bbdd2f7549c8820d61844dae6fc3ef"
server_key_version: "encryptionkey@0"
I20260928 15:51:12.953895 29450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:12.987106 29450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:12.991676 29450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:12.992038 29450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:12.993191 29450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:12.994678 29450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:12.994788 29450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:12.994953 29450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:12.995038 29450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:13.031489 29450 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:38803
I20260928 15:51:13.031533 29578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:38803 every 8 connection(s)
I20260928 15:51:13.032919 29450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:13.038862 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 29450
I20260928 15:51:13.039096 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:51:13.039955 23159 external_mini_cluster.cc:1428] Setting key 27626b2e03b60536654fa94785094b8a
Sep 28 15:51:13 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610672, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:13.058811 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:13.035847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:58301 (local address 127.22.157.254:46257)
0928 15:51:13.036321 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:13.036326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:13.037256 (+ 930us) server_negotiation.cc:407] Connection header received
0928 15:51:13.039652 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:13.039656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:13.039742 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:13.039845 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:13.041732 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:13.042652 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:13.045534 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:13.045953 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:13.046060 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:13.050993 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:13.051001 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:13.051009 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:13.051058 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:13.054943 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:13.055813 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:13.055821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:13.055828 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:13.056012 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:13.056476 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:13.056484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:13.056490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:13.056925 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:13.057101 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:13.057983 (+ 882us) server_negotiation.cc:299] Negotiation successful
0928 15:51:13.058280 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
I20260928 15:51:13.059760 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:13.036326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.193:58301)
0928 15:51:13.037093 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:13.037132 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:13.039278 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:13.040172 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:13.040185 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:13.040942 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:13.041560 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:13.041579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:13.042844 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:13.042852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:13.045329 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:13.045349 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:13.046229 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:13.046239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:13.046600 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:13.047416 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:13.047461 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:13.050675 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:13.055159 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:13.055167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:13.055182 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:13.055660 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:13.056153 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:13.056156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:13.056159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:13.056350 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:13.057066 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:13.057072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:13.057472 (+ 400us) client_negotiation.cc:769] Sending connection context
0928 15:51:13.057847 (+ 375us) client_negotiation.cc:240] Negotiation successful
0928 15:51:13.058315 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":352,"threads_started":1}
I20260928 15:51:13.061575 29579 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46257
I20260928 15:51:13.061920 29579 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:13.062698 29579 heartbeater.cc:511] Master 127.22.157.254:46257 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.22.157.194@KRBTEST.COM; defaulting to no policy
I20260928 15:51:13.065199 29388 ts_manager.cc:194] Registered new tserver with Master: d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803)
I20260928 15:51:13.066852 29388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:58301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab.
Entry for principal oryx/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab.
I20260928 15:51:13.096585 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46257
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab
--principal=oryx/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:13.436914 29585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:13.437232 29585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:13.437414 29585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:13.445178 29585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:13.445319 29585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:13.445511 29585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:51:13.456017 29585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab
--principal=oryx/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:13.457541 29585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:13.459139 29585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:13.474241 29591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:13.474354 29590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:13.474376 29593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:13.475677 29585 server_base.cc:1034] running on GCE node
I20260928 15:51:13.476554 29585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:13.477701 29585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:13.478966 29585 hybrid_clock.cc:648] HybridClock initialized: now 1790610673478896 us; error 61 us; skew 500 ppm
I20260928 15:51:13.479341 29585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:13 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610673, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:13.483167 29585 init.cc:377] Logged in from keytab as oryx/127.22.157.194@KRBTEST.COM (short username oryx)
I20260928 15:51:13.484819 29585 webserver.cc:460] Webserver started at http://127.22.157.194:45755/ using document root <none> and password file <none>
I20260928 15:51:13.485577 29585 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:13.485692 29585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:13.485962 29585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:13.489212 29585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "0bda64e44c8f4422ad664c137df59822"
format_stamp: "Formatted at 2026-09-28 15:51:13 on dist-test-slave-1vw0"
server_key: "2a76056a7ddd0f308413db5e1e2ee11c"
server_key_iv: "8df70c1131d672f396dcceb8089141b6"
server_key_version: "encryptionkey@0"
I20260928 15:51:13.490016 29585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "0bda64e44c8f4422ad664c137df59822"
format_stamp: "Formatted at 2026-09-28 15:51:13 on dist-test-slave-1vw0"
server_key: "2a76056a7ddd0f308413db5e1e2ee11c"
server_key_iv: "8df70c1131d672f396dcceb8089141b6"
server_key_version: "encryptionkey@0"
I20260928 15:51:13.495738 29585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:51:13.499675 29600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:13.501384 29585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:13.501542 29585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "0bda64e44c8f4422ad664c137df59822"
format_stamp: "Formatted at 2026-09-28 15:51:13 on dist-test-slave-1vw0"
server_key: "2a76056a7ddd0f308413db5e1e2ee11c"
server_key_iv: "8df70c1131d672f396dcceb8089141b6"
server_key_version: "encryptionkey@0"
I20260928 15:51:13.501766 29585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:13.519806 29585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:13.525313 29585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:13.525615 29585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:13.526608 29585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:13.528158 29585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:13.528257 29585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:13.528348 29585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:13.528394 29585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:13.563872 29585 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:32891
I20260928 15:51:13.563921 29713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:32891 every 8 connection(s)
I20260928 15:51:13.565400 29585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:51:13.571125 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 29585
I20260928 15:51:13.571336 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:51:13.571894 23159 external_mini_cluster.cc:1428] Setting key 005c2f4057f7251aae39f1743404cb36
Sep 28 15:51:13 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610673, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:13.590502 29716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:13.568320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:45777 (local address 127.22.157.254:46257)
0928 15:51:13.568870 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:13.568875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:13.569885 (+ 1010us) server_negotiation.cc:407] Connection header received
0928 15:51:13.572575 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:13.572580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:13.572700 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:13.572852 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:13.574905 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:13.575640 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:13.578410 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:13.578845 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:13.578966 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:13.583645 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:13.583652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:13.583674 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:13.583726 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:13.586545 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:13.587765 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:13.587770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:13.587772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:13.587918 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:13.588497 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:13.588501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:13.588503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:13.588881 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:13.589022 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:13.590019 (+ 997us) server_negotiation.cc:299] Negotiation successful
0928 15:51:13.590226 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
I20260928 15:51:13.592295 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:13.568942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:45777)
0928 15:51:13.569714 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:13.569755 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:13.572185 (+ 2430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:13.573170 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:13.573182 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:13.573788 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:13.574707 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:13.574732 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:13.575819 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:13.575828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:13.578193 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:13.578215 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:13.579128 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:13.579137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:13.579530 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:13.580375 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:13.580414 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:13.583333 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:13.586751 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:13.586766 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:13.586791 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:13.587562 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:13.588058 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:13.588065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:13.588072 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:13.588362 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:13.589026 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:13.589037 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:13.589581 (+ 544us) client_negotiation.cc:769] Sending connection context
0928 15:51:13.589916 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 15:51:13.590623 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":286,"threads_started":1}
I20260928 15:51:13.594287 29714 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46257
I20260928 15:51:13.594666 29714 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:13.595482 29714 heartbeater.cc:511] Master 127.22.157.254:46257 requested a full tablet report, sending...
I20260928 15:51:13.597177 29388 ts_manager.cc:194] Registered new tserver with Master: 0bda64e44c8f4422ad664c137df59822 (127.22.157.194:32891)
WARNING: no policy specified for oryx/127.22.157.195@KRBTEST.COM; defaulting to no policy
I20260928 15:51:13.598104 29388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:45777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab.
Entry for principal oryx/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab.
I20260928 15:51:13.628573 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46257
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab
--principal=oryx/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:13.974217 29720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:13.974596 29720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:13.974781 29720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:13.982570 29720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:13.982748 29720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:13.982944 29720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:51:13.993314 29720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab
--principal=oryx/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:13.994892 29720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:13.996357 29720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:14.010349 29726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:14.010349 29725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:14.010685 29728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:14.011641 29720 server_base.cc:1034] running on GCE node
I20260928 15:51:14.012415 29720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:14.013536 29720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:14.014751 29720 hybrid_clock.cc:648] HybridClock initialized: now 1790610674014707 us; error 43 us; skew 500 ppm
I20260928 15:51:14.015054 29720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:14 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610674, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:14.018534 29720 init.cc:377] Logged in from keytab as oryx/127.22.157.195@KRBTEST.COM (short username oryx)
I20260928 15:51:14.020205 29720 webserver.cc:460] Webserver started at http://127.22.157.195:41951/ using document root <none> and password file <none>
I20260928 15:51:14.021026 29720 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:14.021202 29720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:14.021523 29720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:14.024660 29720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2"
format_stamp: "Formatted at 2026-09-28 15:51:14 on dist-test-slave-1vw0"
server_key: "dec6f44d02bacb7490d0202336d6866e"
server_key_iv: "891df3208080f5f0e6dbf1139cad5af7"
server_key_version: "encryptionkey@0"
I20260928 15:51:14.025691 29720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2"
format_stamp: "Formatted at 2026-09-28 15:51:14 on dist-test-slave-1vw0"
server_key: "dec6f44d02bacb7490d0202336d6866e"
server_key_iv: "891df3208080f5f0e6dbf1139cad5af7"
server_key_version: "encryptionkey@0"
I20260928 15:51:14.031702 29720 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:51:14.036548 29735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:14.038194 29720 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:51:14.038347 29720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2"
format_stamp: "Formatted at 2026-09-28 15:51:14 on dist-test-slave-1vw0"
server_key: "dec6f44d02bacb7490d0202336d6866e"
server_key_iv: "891df3208080f5f0e6dbf1139cad5af7"
server_key_version: "encryptionkey@0"
I20260928 15:51:14.038529 29720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:14.061971 29720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:14.066262 29720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:14.066526 29720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:14.067469 29720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:14.069022 29720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:14.069108 29720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:14.069202 29720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:14.069247 29720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:14.070520 29579 heartbeater.cc:503] Master 127.22.157.254:46257 was elected leader, sending a full tablet report...
I20260928 15:51:14.105013 29720 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:45159
I20260928 15:51:14.105062 29847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:45159 every 8 connection(s)
I20260928 15:51:14.106596 29720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:51:14.113289 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 29720
I20260928 15:51:14.113494 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:51:14.114066 23159 external_mini_cluster.cc:1428] Setting key f4ecde672890e15ebafa0a091cfcac44
Sep 28 15:51:14 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610674, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:14.131100 29850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.109438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:57359 (local address 127.22.157.254:46257)
0928 15:51:14.109885 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:14.109889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:14.111060 (+ 1171us) server_negotiation.cc:407] Connection header received
0928 15:51:14.113190 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:14.113196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:14.113313 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:14.113449 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:14.115983 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.117303 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.119449 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.119890 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.120009 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.123927 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:14.123934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:14.123938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:14.123981 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:14.126822 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.127787 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.127793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.127795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.127951 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.128607 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.128613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.128634 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.129018 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:14.129229 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:14.130304 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0928 15:51:14.130696 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260928 15:51:14.132354 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.110123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.195:57359)
0928 15:51:14.110874 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:14.110924 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:14.112875 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:14.113824 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:14.113840 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:14.114748 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:14.115700 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.115729 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.117596 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.117602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.119265 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.119281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.120125 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.120129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.120346 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.121164 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:14.121189 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:14.123642 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:14.127040 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.127049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.127066 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.127636 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.128111 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.128116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.128119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.128425 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.129211 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:14.129219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:14.129757 (+ 538us) client_negotiation.cc:769] Sending connection context
0928 15:51:14.130153 (+ 396us) client_negotiation.cc:240] Negotiation successful
0928 15:51:14.130792 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":287,"threads_started":1}
I20260928 15:51:14.134101 29848 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46257
I20260928 15:51:14.134488 29848 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:14.135349 29848 heartbeater.cc:511] Master 127.22.157.254:46257 requested a full tablet report, sending...
I20260928 15:51:14.137171 29388 ts_manager.cc:194] Registered new tserver with Master: d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159)
I20260928 15:51:14.138098 29388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:57359
I20260928 15:51:14.143662 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.157.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.253.keytab.
Entry for principal oryx/127.22.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.253.keytab.
I20260928 15:51:14.196964 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.253:45249
--webserver_interface=127.22.157.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.253.keytab
--principal=oryx/127.22.157.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:46257,127.22.157.253:45249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:14.539503 29854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:14.539884 29854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:14.540042 29854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:14.547894 29854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:14.548079 29854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:14.548199 29854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:14.548306 29854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:14.548383 29854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:14.559181 29854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:46257,127.22.157.253:45249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.253.keytab
--principal=oryx/127.22.157.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.253:45249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.157.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
W20260928 15:51:14.559630 29854 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.157.254:46257,127.22.157.253:45249'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 15:51:14.560802 29854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:14.562157 29854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:14.574836 29859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:14.575038 29860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:14.575030 29862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:14.576082 29854 server_base.cc:1034] running on GCE node
I20260928 15:51:14.576944 29854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:14.578441 29854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:14.579677 29854 hybrid_clock.cc:648] HybridClock initialized: now 1790610674579606 us; error 42 us; skew 500 ppm
I20260928 15:51:14.579994 29854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:14 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610674, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:14.588157 29854 init.cc:377] Logged in from keytab as oryx/127.22.157.253@KRBTEST.COM (short username oryx)
I20260928 15:51:14.589994 29854 webserver.cc:460] Webserver started at http://127.22.157.253:45505/ using document root <none> and password file <none>
I20260928 15:51:14.590790 29854 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:14.590948 29854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:14.591326 29854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:14.594911 29854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data/instance:
uuid: "e588dbb8beb24471ac35e7ee90a05417"
format_stamp: "Formatted at 2026-09-28 15:51:14 on dist-test-slave-1vw0"
server_key: "89c1f5ed88c15a5a569000a5333c8e11"
server_key_iv: "5e5752f474eec4290a133502809dcd39"
server_key_version: "encryptionkey@0"
I20260928 15:51:14.595916 29854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal/instance:
uuid: "e588dbb8beb24471ac35e7ee90a05417"
format_stamp: "Formatted at 2026-09-28 15:51:14 on dist-test-slave-1vw0"
server_key: "89c1f5ed88c15a5a569000a5333c8e11"
server_key_iv: "5e5752f474eec4290a133502809dcd39"
server_key_version: "encryptionkey@0"
I20260928 15:51:14.601558 29714 heartbeater.cc:503] Master 127.22.157.254:46257 was elected leader, sending a full tablet report...
I20260928 15:51:14.602097 29854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260928 15:51:14.606621 29869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:14.608243 29854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:51:14.608497 29854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal
uuid: "e588dbb8beb24471ac35e7ee90a05417"
format_stamp: "Formatted at 2026-09-28 15:51:14 on dist-test-slave-1vw0"
server_key: "89c1f5ed88c15a5a569000a5333c8e11"
server_key_iv: "5e5752f474eec4290a133502809dcd39"
server_key_version: "encryptionkey@0"
I20260928 15:51:14.608774 29854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:14.638305 29854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:14.642858 29854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:14.643105 29854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:14.662456 29854 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.253:45249
I20260928 15:51:14.662431 29922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.253:45249 every 8 connection(s)
I20260928 15:51:14.664109 29854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
I20260928 15:51:14.668071 29923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:14.671343 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 29854
I20260928 15:51:14.675978 29923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:51:14 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.253@KRBTEST.COM
Sep 28 15:51:14 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610674, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.253@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:14.695540 29927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.679104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.22.157.254:46257)
0928 15:51:14.679663 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:14.679669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:14.680191 (+ 522us) server_negotiation.cc:407] Connection header received
0928 15:51:14.680388 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:14.680393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:14.680473 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:14.680564 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:14.681975 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.683051 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.685155 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.685820 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.685961 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.689663 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:14.689668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:14.689672 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:14.689718 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:14.692174 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.692993 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.693001 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.693008 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.693180 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.693650 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.693655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.693659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.694012 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:14.694195 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:14.694884 (+ 689us) server_negotiation.cc:299] Negotiation successful
0928 15:51:14.695150 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":263,"threads_started":1}
I20260928 15:51:14.696928 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.679131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:50810)
0928 15:51:14.680034 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:14.680077 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:14.680254 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:14.680676 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:14.680683 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:14.681579 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:14.681821 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.681830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.683211 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.683217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.684965 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.684981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.686052 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.686058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.686182 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.686829 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:14.686853 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:14.689444 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:14.692313 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.692325 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.692340 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.692850 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.693312 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.693319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.693325 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.693525 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.694160 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:14.694168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:14.694390 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 15:51:14.694758 (+ 368us) client_negotiation.cc:240] Negotiation successful
0928 15:51:14.695935 (+ 1177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":343,"thread_start_us":495,"threads_started":1}
I20260928 15:51:14.696933 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.673826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.22.157.253:45249)
0928 15:51:14.674653 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:14.674666 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:14.674710 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:51:14.676394 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:14.676418 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:14.676969 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:14.677382 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:14.679159 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.681358 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.683078 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.684192 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.684380 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.688985 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:14.688997 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:14.689015 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:14.689060 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:14.692265 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.693081 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.693089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.693091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.693289 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.693679 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.693682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.693684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.694382 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:14.694764 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:14.695033 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 15:51:14.695389 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":607,"thread_start_us":372,"threads_started":1}
I20260928 15:51:14.698941 29923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:51:14 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610674, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.253@KRBTEST.COM for oryx/127.22.157.253@KRBTEST.COM
I20260928 15:51:14.712245 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.699804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.22.157.253:45249)
0928 15:51:14.699979 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:14.699984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:14.700057 (+ 73us) server_negotiation.cc:407] Connection header received
0928 15:51:14.700105 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:14.700109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:14.700179 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:14.700294 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:14.701015 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.701721 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.702886 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.703285 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.703423 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.706973 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:14.706978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:14.706982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:14.707014 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:14.709766 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.710470 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.710474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.710476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.710600 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.710926 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.710930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.710932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.711234 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:14.711344 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:14.711667 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 15:51:14.711928 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":53}
I20260928 15:51:14.712239 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.699645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.0.0.1:38258)
0928 15:51:14.699812 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:14.699827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:14.699928 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:14.700310 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:14.700314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:14.700700 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:14.700894 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.700902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.701862 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.701867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.702746 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.702757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.703433 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.703438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.703713 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.704258 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:14.704274 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:14.706694 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:14.709908 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.709912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.709915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.710304 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.710693 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.710696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.710698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.710827 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.711332 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:14.711335 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:14.711458 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 15:51:14.711577 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 15:51:14.711812 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":39}
I20260928 15:51:14.717573 29923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Bootstrap starting.
I20260928 15:51:14.720923 29923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:14.721998 29923 log.cc:824] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:14.725116 29923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: No bootstrap required, opened a new log
I20260928 15:51:14.732717 29923 raft_consensus.cc:348] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } }
I20260928 15:51:14.733203 29923 raft_consensus.cc:374] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:14.733302 29923 raft_consensus.cc:729] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e588dbb8beb24471ac35e7ee90a05417, State: Initialized, Role: FOLLOWER
I20260928 15:51:14.733951 29923 consensus_queue.cc:260] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } }
I20260928 15:51:14.734877 29930 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } } }
I20260928 15:51:14.735265 29930 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:51:14.736050 29923 sys_catalog.cc:564] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:51:14.745519 29941 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:51:14.745702 29941 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:51:14.747994 29854 master_runner.cc:428] Detected that this master e588dbb8beb24471ac35e7ee90a05417 is joining an existing cluster
I20260928 15:51:14.748133 29854 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.157.253:45249
I20260928 15:51:14.773993 29927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.753157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.22.157.254:46257)
0928 15:51:14.753392 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:14.753400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:14.757574 (+ 4174us) server_negotiation.cc:407] Connection header received
0928 15:51:14.757790 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:14.757796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:14.757897 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:14.758022 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:14.759333 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.760100 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.762115 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.762897 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.763026 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.766145 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:14.766154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:14.766161 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:14.766214 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:14.770110 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.771039 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.771047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.771050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.771241 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.771758 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.771765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.771769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.772234 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:14.772412 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:14.773259 (+ 847us) server_negotiation.cc:299] Negotiation successful
0928 15:51:14.773558 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:51:14.776062 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.753802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:50824)
0928 15:51:14.757435 (+ 3633us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:14.757452 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:14.757620 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:14.758054 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:14.758060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:14.758798 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:14.759141 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.759152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.760257 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.760264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.761885 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.761911 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.763418 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.763426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.763659 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.764462 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:14.764491 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:14.765884 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:14.770296 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.770309 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.770314 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.770867 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.771414 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.771420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.771425 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.771610 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.772508 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:14.772513 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:14.772834 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 15:51:14.775021 (+ 2187us) client_negotiation.cc:240] Negotiation successful
0928 15:51:14.775471 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3495,"mutex_wait_us":1,"thread_start_us":3138,"threads_started":1}
WARNING: no policy specified for oryx/127.22.157.252@KRBTEST.COM; defaulting to no policy
I20260928 15:51:14.780782 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.761038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.22.157.253:45249)
0928 15:51:14.761266 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:14.761273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:14.761294 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:51:14.761348 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:14.761354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:14.761460 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:14.761570 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:14.763187 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.764366 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.766594 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.767388 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.767502 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.771304 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:14.771313 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:14.771317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:14.771355 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:14.775447 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.777930 (+ 2483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.777937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.777942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.778135 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.778734 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.778740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.778743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.779184 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:14.779348 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:14.779915 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 15:51:14.780158 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.252@KRBTEST.COM" created.
I20260928 15:51:14.782505 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.753294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.0.0.1:38262)
0928 15:51:14.755612 (+ 2318us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:14.755642 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:14.755804 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:14.761752 (+ 5948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:14.761758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:14.762586 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:14.763018 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.763033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.764624 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.764635 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.766432 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.766449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.768965 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.768978 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.769189 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.769904 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:14.769940 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:14.771116 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:14.775925 (+ 4809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.775930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.775934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.777767 (+ 1833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.778375 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.778380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.778384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.778590 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.779424 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:14.779428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:14.779579 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 15:51:14.780798 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0928 15:51:14.781947 (+ 1149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"thread_start_us":1955,"threads_started":1}
Sep 28 15:51:14 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610672, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.254@KRBTEST.COM for oryx/127.22.157.253@KRBTEST.COM
I20260928 15:51:14.805671 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.784125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.22.157.253:45249)
0928 15:51:14.784311 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:14.784318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:14.785140 (+ 822us) server_negotiation.cc:407] Connection header received
0928 15:51:14.785557 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:14.785564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:14.785668 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:14.785810 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:14.787180 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.787962 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.790747 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.791564 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.791698 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.796968 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:14.796980 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:14.796989 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:14.797043 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:14.801713 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.802932 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.802941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.802947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.803148 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.803656 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.803664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.803669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.804062 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:14.804244 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:14.804999 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 15:51:14.805258 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:51:14.805692 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.784197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.0.0.1:38268)
0928 15:51:14.785000 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:14.785041 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:14.785286 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:14.786010 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:14.786022 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:14.786723 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:14.786978 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.786989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.788124 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.788132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.790563 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.790583 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.791870 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.791883 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.792171 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.793057 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:14.793095 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:14.796622 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:14.801952 (+ 5330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.801973 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.801982 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.802744 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.803307 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.803312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.803317 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.803504 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.804232 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:14.804244 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:14.804520 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 15:51:14.804780 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 15:51:14.805193 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":330,"threads_started":1}
I20260928 15:51:14.808128 29388 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.157.253:45249: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417"
member_type: NON_VOTER
last_known_addr {
host: "127.22.157.253"
port: 45249
}
attrs {
promote: true
}
}
dest_uuid: "4d59f65974d44666b7524a34baf610b7"
cas_config_opid_index: -1
I20260928 15:51:14.809334 29388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } }
I20260928 15:51:14.814313 29954 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e588dbb8beb24471ac35e7ee90a05417 (127.22.157.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } } }
I20260928 15:51:14.815197 29897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.252.keytab.
Entry for principal oryx/127.22.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.252.keytab.
I20260928 15:51:14.819710 29897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Refusing update from remote peer 4d59f65974d44666b7524a34baf610b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 15:51:14.820788 29954 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.157.253:45249
I20260928 15:51:14.821887 29854 master.cc:537] Master@127.22.157.253:45249 shutting down...
I20260928 15:51:14.821877 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.252:44755
--webserver_interface=127.22.157.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.252.keytab
--principal=oryx/127.22.157.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:46257,127.22.157.253:45249,127.22.157.252:44755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:51:14.823347 29930 raft_consensus.cc:482] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:14.823593 29930 raft_consensus.cc:504] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } }
I20260928 15:51:14.824877 29930 leader_election.cc:288] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257)
I20260928 15:51:14.825702 29955 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:14.826874 29398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e588dbb8beb24471ac35e7ee90a05417" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d59f65974d44666b7524a34baf610b7" is_pre_election: true
I20260928 15:51:14.829345 29873 leader_election.cc:302] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: e588dbb8beb24471ac35e7ee90a05417; no voters: 4d59f65974d44666b7524a34baf610b7
W20260928 15:51:14.829694 29374 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 15:51:14.830102 29930 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260928 15:51:14.831470 29374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 -> Peer e588dbb8beb24471ac35e7ee90a05417 (127.22.157.253:45249): Couldn't send request to peer e588dbb8beb24471ac35e7ee90a05417. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 15:51:14.836167 29953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } } }
I20260928 15:51:14.836617 29953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:14.839448 29854 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:14.839932 29854 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:14.840129 29854 tablet_replica.cc:330] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: stopping tablet replica
I20260928 15:51:14.848567 29854 master.cc:552] Master@127.22.157.253:45249 shutdown complete.
I20260928 15:51:14.848665 29854 master_runner.cc:305] Clearing existing system tablet
I20260928 15:51:14.849139 29854 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:14.849288 29854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:14.852406 29854 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 15:51:14.857117 29960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:14.857621 29854 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:14.858112 29854 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:51:14.858256 29854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal
uuid: "e588dbb8beb24471ac35e7ee90a05417"
format_stamp: "Formatted at 2026-09-28 15:51:14 on dist-test-slave-1vw0"
server_key: "89c1f5ed88c15a5a569000a5333c8e11"
server_key_iv: "5e5752f474eec4290a133502809dcd39"
server_key_version: "encryptionkey@0"
I20260928 15:51:14.863829 29854 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:14.871345 29854 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 15:51:14.871685 29854 log.cc:1196] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 15:51:14.872200 29854 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Deleting consensus metadata
I20260928 15:51:14.872572 29854 master_runner.cc:315] Copying system tablet from 127.22.157.254:46257
I20260928 15:51:14.875262 29854 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: tablet copy: Beginning tablet copy session from remote peer at address 127.22.157.254:46257
I20260928 15:51:14.893043 29927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.876365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.22.157.254:46257)
0928 15:51:14.876579 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:14.876587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:14.877285 (+ 698us) server_negotiation.cc:407] Connection header received
0928 15:51:14.877469 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:14.877475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:14.877593 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:14.877714 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:14.878972 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.880133 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.882033 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.882698 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.882837 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.885321 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:14.885328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:14.885333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:14.885376 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:14.889713 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.890624 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.890630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.890633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.890786 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:14.891238 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:14.891243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:14.891246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:14.891697 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:14.891861 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:14.892329 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 15:51:14.892647 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:51:14.893061 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:14.876163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:50838)
0928 15:51:14.877134 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:14.877160 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:14.877319 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:14.877764 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:14.877770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:14.878281 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:14.878718 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.878735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.880296 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.880302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.881859 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:14.881876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:14.882872 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:14.882878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:14.883089 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:14.883849 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:14.883874 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:14.885080 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:14.889890 (+ 4810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.889897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.889902 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.890446 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.890910 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:14.890916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:14.890920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:14.891100 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:14.891842 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:14.891847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:14.892039 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 15:51:14.892187 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 15:51:14.892541 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":702,"threads_started":1}
I20260928 15:51:14.894395 29408 tablet_copy_service.cc:140] P 4d59f65974d44666b7524a34baf610b7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e588dbb8beb24471ac35e7ee90a05417 ({username='oryx', principal='oryx/127.22.157.253@KRBTEST.COM'} at 127.0.0.1:50838)
I20260928 15:51:14.894709 29408 tablet_copy_service.cc:161] P 4d59f65974d44666b7524a34baf610b7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e588dbb8beb24471ac35e7ee90a05417 at {username='oryx', principal='oryx/127.22.157.253@KRBTEST.COM'} at 127.0.0.1:50838: session id = e588dbb8beb24471ac35e7ee90a05417-00000000000000000000000000000000
I20260928 15:51:14.899516 29408 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 15:51:14.902031 29854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 15:51:14.909377 29854 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: tablet copy: Starting download of 0 data blocks...
I20260928 15:51:14.909811 29854 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: tablet copy: Starting download of 1 WAL segments...
I20260928 15:51:14.913014 29854 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 15:51:14.917946 29408 tablet_copy_service.cc:342] P 4d59f65974d44666b7524a34baf610b7: Request end of tablet copy session e588dbb8beb24471ac35e7ee90a05417-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.157.253@KRBTEST.COM'} at 127.0.0.1:50838
I20260928 15:51:14.918187 29408 tablet_copy_service.cc:433] P 4d59f65974d44666b7524a34baf610b7: ending tablet copy session e588dbb8beb24471ac35e7ee90a05417-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e588dbb8beb24471ac35e7ee90a05417
I20260928 15:51:14.931176 29854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:14.934195 29967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:14.938349 29968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:14.942852 29854 server_base.cc:1034] running on GCE node
W20260928 15:51:14.945056 29970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:14.945570 29854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:14.946296 29854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:14.947546 29854 hybrid_clock.cc:648] HybridClock initialized: now 1790610674947477 us; error 42 us; skew 500 ppm
I20260928 15:51:14.947798 29854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:14 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610674, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:14.950147 29854 init.cc:377] Logged in from keytab as oryx/127.22.157.253@KRBTEST.COM (short username oryx)
I20260928 15:51:14.951507 29854 webserver.cc:460] Webserver started at http://127.22.157.253:45057/ using document root <none> and password file <none>
I20260928 15:51:14.951885 29854 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:14.951967 29854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:14.955703 29854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 15:51:14.957913 29977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:14.958412 29854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 15:51:14.958542 29854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal
uuid: "e588dbb8beb24471ac35e7ee90a05417"
format_stamp: "Formatted at 2026-09-28 15:51:14 on dist-test-slave-1vw0"
server_key: "89c1f5ed88c15a5a569000a5333c8e11"
server_key_iv: "5e5752f474eec4290a133502809dcd39"
server_key_version: "encryptionkey@0"
I20260928 15:51:14.958732 29854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:14.978163 29854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:14.982291 29854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:15.009191 29854 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.253:45249
I20260928 15:51:15.009204 30029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.253:45249 every 8 connection(s)
I20260928 15:51:15.010627 29854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
I20260928 15:51:15.022197 30030 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:51:15.026294 30030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Bootstrap starting.
I20260928 15:51:15.043922 30030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:51:15.044415 30030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Bootstrap complete.
I20260928 15:51:15.045359 30030 raft_consensus.cc:348] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } }
I20260928 15:51:15.045598 30030 raft_consensus.cc:729] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e588dbb8beb24471ac35e7ee90a05417, State: Initialized, Role: LEARNER
I20260928 15:51:15.045825 30030 consensus_queue.cc:260] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } }
I20260928 15:51:15.046685 30033 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } } }
I20260928 15:51:15.047034 30030 sys_catalog.cc:564] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:15.047083 30033 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:51:15.055114 30044 catalog_manager.cc:1259] Loaded cluster ID: 5b1dfd5385a4479081dbef524155c4f6
I20260928 15:51:15.055351 30044 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: loading cluster ID for follower catalog manager: success
I20260928 15:51:15.058238 30044 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: acquiring CA information for follower catalog manager: success
I20260928 15:51:15.060621 30044 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:51:15.141150 29848 heartbeater.cc:503] Master 127.22.157.254:46257 was elected leader, sending a full tablet report...
I20260928 15:51:15.280654 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.271787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.0.0.1:38282)
0928 15:51:15.272091 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:15.272114 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:15.272250 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:15.273389 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:15.273394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:15.273428 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:15.273622 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.273635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.275831 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.275837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.277992 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.278010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.279765 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.279772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.280001 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.280079 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 15:51:15.280189 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:51:15.280351 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260928 15:51:15.280800 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.272024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.22.157.253:45249)
0928 15:51:15.272951 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.272958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.272982 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:51:15.273072 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.273078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.273167 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.273302 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:15.273831 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.275651 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.278190 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.279566 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.279782 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.279976 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.280229 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.280380 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":548,"threads_started":1}
I20260928 15:51:15.281996 30005 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 15:51:15.282599 30033 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d59f65974d44666b7524a34baf610b7. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } } }
I20260928 15:51:15.282882 30033 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: This master's current role is: LEARNER
W20260928 15:51:15.284240 29957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:15.284534 29957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:15.284761 29957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:15.292254 29957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:15.292371 29957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:15.292476 29957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:15.292577 29957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:15.292670 29957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:15.303087 29957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:46257,127.22.157.253:45249,127.22.157.252:44755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.252.keytab
--principal=oryx/127.22.157.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.252:44755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.157.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:15.304541 29957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:15.305975 29957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:15.318837 30051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:15.318868 30050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:15.318832 30053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:15.319921 29957 server_base.cc:1034] running on GCE node
I20260928 15:51:15.320699 29957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:15.322311 29957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:15.323515 29957 hybrid_clock.cc:648] HybridClock initialized: now 1790610675323474 us; error 40 us; skew 500 ppm
I20260928 15:51:15.323869 29957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:15 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610675, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:15.327461 29957 init.cc:377] Logged in from keytab as oryx/127.22.157.252@KRBTEST.COM (short username oryx)
I20260928 15:51:15.329210 29957 webserver.cc:460] Webserver started at http://127.22.157.252:35681/ using document root <none> and password file <none>
I20260928 15:51:15.329872 29957 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:15.329972 29957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:15.330235 29957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:15.333323 29957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data/instance:
uuid: "380d381cd4b44b80bafd3a5fc51beb24"
format_stamp: "Formatted at 2026-09-28 15:51:15 on dist-test-slave-1vw0"
server_key: "390021239b3d276ebeaf8d6fd2be00ee"
server_key_iv: "a62e5bab46014c0833ff1cdf0b2ff7e8"
server_key_version: "encryptionkey@0"
I20260928 15:51:15.334069 29957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal/instance:
uuid: "380d381cd4b44b80bafd3a5fc51beb24"
format_stamp: "Formatted at 2026-09-28 15:51:15 on dist-test-slave-1vw0"
server_key: "390021239b3d276ebeaf8d6fd2be00ee"
server_key_iv: "a62e5bab46014c0833ff1cdf0b2ff7e8"
server_key_version: "encryptionkey@0"
I20260928 15:51:15.339522 29957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:51:15.343214 30061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:15.344707 29957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:51:15.344975 29957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal
uuid: "380d381cd4b44b80bafd3a5fc51beb24"
format_stamp: "Formatted at 2026-09-28 15:51:15 on dist-test-slave-1vw0"
server_key: "390021239b3d276ebeaf8d6fd2be00ee"
server_key_iv: "a62e5bab46014c0833ff1cdf0b2ff7e8"
server_key_version: "encryptionkey@0"
I20260928 15:51:15.345178 29957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:15.361007 29957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:15.365549 29957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:15.365810 29957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:15.386883 29957 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.252:44755
I20260928 15:51:15.386933 30113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.252:44755 every 8 connection(s)
I20260928 15:51:15.388306 29957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
I20260928 15:51:15.389046 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 29957
I20260928 15:51:15.393344 30114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:15.401345 30114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:51:15 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.252@KRBTEST.COM
I20260928 15:51:15.413686 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.391611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.22.157.252:44755)
0928 15:51:15.392370 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.392383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.392426 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:51:15.394192 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.394215 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.394592 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.394975 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.396551 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.398049 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.399409 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.400106 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.400308 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.405294 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.405304 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.405322 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.405370 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.408540 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.409489 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.409495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.409501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.409701 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.410188 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.410192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.410194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.410959 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.411335 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.411929 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.412445 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":329,"threads_started":1}
Sep 28 15:51:15 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610675, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.252@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:15.421445 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.404221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.22.157.254:46257)
0928 15:51:15.405131 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.405140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.405643 (+ 503us) server_negotiation.cc:407] Connection header received
0928 15:51:15.405890 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.405896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.405985 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.406111 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.407649 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.408412 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.410448 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.410883 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.411046 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.414945 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.414951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.414954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.415007 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.418183 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.419098 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.419105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.419111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.419251 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.419697 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.419701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.419704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.420033 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.420204 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.420749 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.421012 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":642,"threads_started":1}
I20260928 15:51:15.421485 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.404105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:50844)
0928 15:51:15.405487 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:15.405538 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:15.405749 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:15.406158 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:15.406166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:15.407122 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:15.407478 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.407489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.408565 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.408572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.410302 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.410316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.411093 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.411100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.411301 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.412005 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:15.412025 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:15.414664 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:15.418349 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.418454 (+ 105us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.418460 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.418944 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.419384 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.419390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.419394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.419550 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.420188 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:15.420196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:15.420404 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 15:51:15.420600 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 15:51:15.420994 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":432,"threads_started":1}
I20260928 15:51:15.423153 30114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:51:15 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610675, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.252@KRBTEST.COM for oryx/127.22.157.253@KRBTEST.COM
I20260928 15:51:15.436347 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.423975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.0.0.1:38294)
0928 15:51:15.424217 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:15.424234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:15.424381 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:15.424852 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:15.424856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:15.425193 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:15.425340 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.425348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.426417 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.426422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.427357 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.427376 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.428478 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.428482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.428693 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.429332 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:15.429349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:15.431384 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:15.434328 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.434332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.434335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.434745 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.435073 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.435076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.435078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.435204 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.435656 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:15.435659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:15.435780 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 15:51:15.435888 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:51:15.436115 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260928 15:51:15.436465 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.423957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.22.157.253:45249)
0928 15:51:15.424184 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.424193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.424305 (+ 112us) server_negotiation.cc:407] Connection header received
0928 15:51:15.424594 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.424599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.424665 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.424848 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.425508 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.426279 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.427610 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.428319 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.428438 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.431586 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.431592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.431595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.431630 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.434196 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.434850 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.434854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.434857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.434980 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.435292 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.435295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.435297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.435557 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.435675 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.435977 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.436185 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:51:15.437582 30114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:51:15 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610675, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.252@KRBTEST.COM for oryx/127.22.157.252@KRBTEST.COM
I20260928 15:51:15.450872 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.438247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44755 (local address 127.0.0.1:47524)
0928 15:51:15.438532 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:15.438553 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:15.438655 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:15.438989 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:15.438993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:15.439492 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:15.439684 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.439692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.440690 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.440698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.441605 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.441616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.442423 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.442429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.442620 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.443244 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:15.443261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:15.445268 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:15.448113 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.448120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.448124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.448556 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.448956 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.448963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.448966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.449182 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.449753 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:15.449757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:15.449947 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 15:51:15.450096 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 15:51:15.450392 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 15:51:15.450917 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.438452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.22.157.252:44755)
0928 15:51:15.438609 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.438615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.438631 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:51:15.438795 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.438800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.438878 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.438974 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.439816 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.440573 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.441840 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.442271 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.442408 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.445451 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.445455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.445458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.445486 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.447957 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.448667 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.448674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.448678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.448854 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.449288 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.449293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.449296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.449648 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.449839 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.450211 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.450551 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":52}
I20260928 15:51:15.455979 30114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: Bootstrap starting.
I20260928 15:51:15.459635 30114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:15.460696 30114 log.cc:824] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:15.463598 30114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: No bootstrap required, opened a new log
I20260928 15:51:15.471701 30114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } }
I20260928 15:51:15.472062 30114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:15.472143 30114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380d381cd4b44b80bafd3a5fc51beb24, State: Initialized, Role: FOLLOWER
I20260928 15:51:15.472692 30114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } }
I20260928 15:51:15.473613 30121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } } }
I20260928 15:51:15.473973 30121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:51:15.474720 30114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:51:15.484367 30132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:51:15.484532 30132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:51:15.488893 29957 master_runner.cc:428] Detected that this master 380d381cd4b44b80bafd3a5fc51beb24 is joining an existing cluster
I20260928 15:51:15.489006 29957 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.157.252:44755
I20260928 15:51:15.512317 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.493409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38302 (local address 127.22.157.253:45249)
0928 15:51:15.493585 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.493591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.494247 (+ 656us) server_negotiation.cc:407] Connection header received
0928 15:51:15.494452 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.494458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.494551 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.494660 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.496665 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.498030 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.500148 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.500824 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.500925 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.503939 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.503946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.503952 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.503998 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.508549 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.509564 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.509571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.509574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.509729 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.510251 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.510257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.510260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.510704 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.510831 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.511531 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.511788 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:51:15.515054 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.493393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.0.0.1:38302)
0928 15:51:15.494129 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:15.494150 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:15.494295 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:15.494863 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:15.494869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:15.495666 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:15.496481 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.496494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.498233 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.498238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.499948 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.499964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.501308 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.501316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.501589 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.502396 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:15.502420 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:15.503721 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:15.508812 (+ 5091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.508822 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.508828 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.509401 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.509877 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.509883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.509891 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.510115 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.510926 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:15.510931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:15.511222 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 15:51:15.512601 (+ 1379us) client_negotiation.cc:240] Negotiation successful
0928 15:51:15.514382 (+ 1781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":36,"thread_start_us":411,"threads_started":1}
I20260928 15:51:15.515991 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.494311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44755 (local address 127.0.0.1:47540)
0928 15:51:15.494899 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:15.494916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:15.495065 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:15.495634 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:15.495640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:15.496554 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:15.497271 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.497290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.499279 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.499287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.501565 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.501583 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.503227 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.503235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.503507 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.504372 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:15.504427 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:15.506235 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:15.512017 (+ 5782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.512022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.512027 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.513376 (+ 1349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.513818 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.513823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.513827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.513985 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.514852 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:15.514857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:15.515030 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 15:51:15.515216 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 15:51:15.515518 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":25,"thread_start_us":569,"threads_started":1}
I20260928 15:51:15.516942 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.494691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.22.157.252:44755)
0928 15:51:15.494887 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.494893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.494985 (+ 92us) server_negotiation.cc:407] Connection header received
0928 15:51:15.495247 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.495255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.495343 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.495422 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.497461 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.499016 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.501740 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.502714 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.502833 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.506490 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.506503 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.506511 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.506558 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.511826 (+ 5268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.513516 (+ 1690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.513520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.513523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.513692 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.514136 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.514140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.514144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.514718 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.514963 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.516319 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.516546 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:51:15.519995 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 29450
I20260928 15:51:15.530002 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.495426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:50850)
0928 15:51:15.515300 (+ 19874us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:15.515319 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:15.515523 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:15.516068 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:15.516086 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:15.516804 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:15.517176 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.517188 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.518696 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.518708 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.520186 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.520203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.521493 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.521504 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.521684 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.522339 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:15.522363 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:15.523467 (+ 1104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:15.527071 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.527076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.527079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.527568 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.528049 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.528053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.528057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.528252 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.529036 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:15.529039 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:15.529262 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 15:51:15.529428 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 15:51:15.529688 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19727,"thread_start_us":18758,"threads_started":1}
I20260928 15:51:15.530754 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.493174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.22.157.254:46257)
0928 15:51:15.493408 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.493415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.515410 (+ 21995us) server_negotiation.cc:407] Connection header received
0928 15:51:15.515698 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.515705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.515794 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.515865 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.517358 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.518531 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.520357 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.521126 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.521257 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.523678 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.523687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.523696 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.523741 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.526910 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.527734 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.527740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.527744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.527914 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.528393 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.528398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.528401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.528845 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.528986 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.530336 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.530513 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 15:51:15 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.193@KRBTEST.COM
I20260928 15:51:15.543316 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.521190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.22.157.193:38803)
0928 15:51:15.521880 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.521887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.521912 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:51:15.522005 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.522014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.522319 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.522495 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.523782 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.525416 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.527384 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.528384 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.528579 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.534742 (+ 6163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.534761 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.534781 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.534830 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.538942 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.540021 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.540032 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.540036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.540235 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.540932 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.540943 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.540947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.541496 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.541951 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.542422 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.542782 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":368,"threads_started":1}
Sep 28 15:51:15 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610672, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.254@KRBTEST.COM for oryx/127.22.157.252@KRBTEST.COM
I20260928 15:51:15.544804 29574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:15.552165 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.536229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.22.157.252:44755)
0928 15:51:15.536387 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.536391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.536404 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:51:15.536518 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.536523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.536608 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.536724 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.537522 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.538275 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.539488 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.540062 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.540207 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.544852 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.544862 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.544867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.544915 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.549240 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.550139 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.550149 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.550153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.550284 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.550701 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.550708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.550711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.551029 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.551190 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.551590 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.551796 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:51:15.552183 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.535875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44755 (local address 127.0.0.1:47548)
0928 15:51:15.536048 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:15.536068 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:15.536173 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:15.536805 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:15.536810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:15.537191 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:15.537352 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.537361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.538410 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.538416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.539296 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.539309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.540258 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.540267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.540607 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.541347 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:15.541381 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:15.544598 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:15.549412 (+ 4814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.549417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.549421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.549988 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.550403 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.550408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.550412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.550594 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.551164 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:15.551169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:15.551336 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 15:51:15.551481 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 15:51:15.551792 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 15:51:15.553812 29388 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.157.252:44755: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24"
member_type: NON_VOTER
last_known_addr {
host: "127.22.157.252"
port: 44755
}
attrs {
promote: true
}
}
dest_uuid: "4d59f65974d44666b7524a34baf610b7"
cas_config_opid_index: 5
I20260928 15:51:15.554648 29388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } }
I20260928 15:51:15.557785 30005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 LEARNER]: Refusing update from remote peer 4d59f65974d44666b7524a34baf610b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 15:51:15.559654 30088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:15.556540 29953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:15.565155 30144 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 15:51:15.571630 30088 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Refusing update from remote peer 4d59f65974d44666b7524a34baf610b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 15:51:15.572939 29953 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.157.252:44755
I20260928 15:51:15.572925 30144 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:15.573820 29957 master.cc:537] Master@127.22.157.252:44755 shutting down...
I20260928 15:51:15.587477 30005 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
W20260928 15:51:15.589922 29374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 -> Peer 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252:44755): Couldn't send request to peer 380d381cd4b44b80bafd3a5fc51beb24. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 15:51:15.592036 30144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:15.592365 30144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:15.593189 30033 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: NON_VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: true } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:15.593544 30033 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:51:15.593904 30144 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: attempting to promote NON_VOTER e588dbb8beb24471ac35e7ee90a05417 to VOTER
I20260928 15:51:15.594839 30144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } }
I20260928 15:51:15.596897 30005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 LEARNER]: Refusing update from remote peer 4d59f65974d44666b7524a34baf610b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 15:51:15.597704 30143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 15:51:15.598238 29374 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 -> Peer 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252:44755): Couldn't send request to peer 380d381cd4b44b80bafd3a5fc51beb24. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 15:51:15.600208 30144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e588dbb8beb24471ac35e7ee90a05417 (127.22.157.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:15.603215 30143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:15.603462 30143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:15.604033 30005 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e588dbb8beb24471ac35e7ee90a05417 (127.22.157.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:15.607858 30143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:15.608198 30143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:15.608238 30033 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:15.608479 30033 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:51:15.615983 29957 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:15.616400 29957 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:15.616626 29957 tablet_replica.cc:330] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: stopping tablet replica
I20260928 15:51:15.623281 29957 master.cc:552] Master@127.22.157.252:44755 shutdown complete.
I20260928 15:51:15.623374 29957 master_runner.cc:305] Clearing existing system tablet
I20260928 15:51:15.623773 29957 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:15.623888 29957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:15.627002 29957 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:51:15.628997 30149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:15.629424 29957 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:15.629874 29957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 15:51:15.630033 29957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal
uuid: "380d381cd4b44b80bafd3a5fc51beb24"
format_stamp: "Formatted at 2026-09-28 15:51:15 on dist-test-slave-1vw0"
server_key: "390021239b3d276ebeaf8d6fd2be00ee"
server_key_iv: "a62e5bab46014c0833ff1cdf0b2ff7e8"
server_key_version: "encryptionkey@0"
I20260928 15:51:15.635452 29957 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:15.642781 29957 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 15:51:15.643074 29957 log.cc:1196] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 15:51:15.643725 29957 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: Deleting consensus metadata
I20260928 15:51:15.644069 29957 master_runner.cc:315] Copying system tablet from 127.22.157.254:46257
I20260928 15:51:15.647595 29957 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: tablet copy: Beginning tablet copy session from remote peer at address 127.22.157.254:46257
I20260928 15:51:15.665560 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.648677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.22.157.254:46257)
0928 15:51:15.648942 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:15.648949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:15.649524 (+ 575us) server_negotiation.cc:407] Connection header received
0928 15:51:15.649814 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:15.649821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:15.649923 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:15.649997 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:15.651359 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.652502 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.654515 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.655279 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.655383 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.658229 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:15.658237 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:15.658242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:15.658291 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:15.662093 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.663022 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.663030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.663035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.663236 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:15.663722 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:15.663728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:15.663731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:15.664114 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:15.664262 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:15.664906 (+ 644us) server_negotiation.cc:299] Negotiation successful
0928 15:51:15.665163 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260928 15:51:15.666954 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:15.648611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:50852)
0928 15:51:15.649377 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:15.649403 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:15.649652 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:15.650237 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:15.650242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:15.650877 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:15.651170 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.651183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.652712 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.652737 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.654322 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:15.654338 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:15.655707 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:15.655713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:15.655928 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:15.656769 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:15.656799 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:15.657953 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:15.662318 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.662324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.662329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.662873 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.663412 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:15.663417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:15.663421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:15.663587 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:15.664343 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:15.664348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:15.664528 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 15:51:15.666121 (+ 1593us) client_negotiation.cc:240] Negotiation successful
0928 15:51:15.666463 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":430,"threads_started":1}
I20260928 15:51:15.667838 29408 tablet_copy_service.cc:140] P 4d59f65974d44666b7524a34baf610b7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 380d381cd4b44b80bafd3a5fc51beb24 ({username='oryx', principal='oryx/127.22.157.252@KRBTEST.COM'} at 127.0.0.1:50852)
I20260928 15:51:15.668009 29408 tablet_copy_service.cc:161] P 4d59f65974d44666b7524a34baf610b7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 380d381cd4b44b80bafd3a5fc51beb24 at {username='oryx', principal='oryx/127.22.157.252@KRBTEST.COM'} at 127.0.0.1:50852: session id = 380d381cd4b44b80bafd3a5fc51beb24-00000000000000000000000000000000
I20260928 15:51:15.671705 29408 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 15:51:15.673813 29957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 15:51:15.682521 29957 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: tablet copy: Starting download of 0 data blocks...
I20260928 15:51:15.682860 29957 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: tablet copy: Starting download of 1 WAL segments...
I20260928 15:51:15.685900 29957 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 15:51:15.686380 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 29450
I20260928 15:51:15.690673 29408 tablet_copy_service.cc:342] P 4d59f65974d44666b7524a34baf610b7: Request end of tablet copy session 380d381cd4b44b80bafd3a5fc51beb24-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.157.252@KRBTEST.COM'} at 127.0.0.1:50852
I20260928 15:51:15.690829 29408 tablet_copy_service.cc:433] P 4d59f65974d44666b7524a34baf610b7: ending tablet copy session 380d381cd4b44b80bafd3a5fc51beb24-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 380d381cd4b44b80bafd3a5fc51beb24
I20260928 15:51:15.705237 29957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:15.708335 30156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:15.710711 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:38803
--local_ip_for_outbound_sockets=127.22.157.193
--tserver_master_addrs=127.22.157.254:46257
--webserver_port=34345
--webserver_interface=127.22.157.193
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab
--principal=oryx/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.157.254:46257,127.22.157.253:45249
--tserver_master_addrs=127.22.157.254:46257,127.22.157.253:45249,127.22.157.252:44755 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:15.710783 30157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:15.721422 29957 server_base.cc:1034] running on GCE node
W20260928 15:51:15.722569 30159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:15.723044 29957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:15.723731 29957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:15.724928 29957 hybrid_clock.cc:648] HybridClock initialized: now 1790610675724873 us; error 46 us; skew 500 ppm
I20260928 15:51:15.725179 29957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:15 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610675, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:15.727147 29957 init.cc:377] Logged in from keytab as oryx/127.22.157.252@KRBTEST.COM (short username oryx)
I20260928 15:51:15.728684 29957 webserver.cc:460] Webserver started at http://127.22.157.252:35599/ using document root <none> and password file <none>
I20260928 15:51:15.729163 29957 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:15.729274 29957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:15.731595 29957 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 15:51:15.733256 30167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:15.733734 29957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 15:51:15.733861 29957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal
uuid: "380d381cd4b44b80bafd3a5fc51beb24"
format_stamp: "Formatted at 2026-09-28 15:51:15 on dist-test-slave-1vw0"
server_key: "390021239b3d276ebeaf8d6fd2be00ee"
server_key_iv: "a62e5bab46014c0833ff1cdf0b2ff7e8"
server_key_version: "encryptionkey@0"
I20260928 15:51:15.734061 29957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:15.746098 29957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:15.750098 29957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:15.774703 29957 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.252:44755
I20260928 15:51:15.774708 30219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.252:44755 every 8 connection(s)
I20260928 15:51:15.775688 29957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
I20260928 15:51:15.788938 30220 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:51:15.792640 30220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: Bootstrap starting.
I20260928 15:51:15.807893 30220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:51:15.808393 30220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: Bootstrap complete.
I20260928 15:51:15.809370 30220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } }
I20260928 15:51:15.809623 30220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 380d381cd4b44b80bafd3a5fc51beb24, State: Initialized, Role: LEARNER
I20260928 15:51:15.809856 30220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } }
I20260928 15:51:15.810702 30223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:15.811029 30220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:15.811165 30223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: This master's current role is: LEARNER
I20260928 15:51:15.824661 30235 catalog_manager.cc:1259] Loaded cluster ID: 5b1dfd5385a4479081dbef524155c4f6
I20260928 15:51:15.824939 30235 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: loading cluster ID for follower catalog manager: success
I20260928 15:51:15.828600 30235 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: acquiring CA information for follower catalog manager: success
I20260928 15:51:15.831084 30235 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:51:16.064503 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.053596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44755 (local address 127.0.0.1:47564)
0928 15:51:16.055699 (+ 2103us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:16.055721 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:16.055894 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:16.056771 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:16.056777 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:16.056809 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:16.057060 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.057071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.059429 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.059483 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.061920 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.061947 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.063838 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.063843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.063994 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.064031 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:51:16.064109 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:51:16.064219 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1977,"thread_start_us":1943,"threads_started":1}
I20260928 15:51:16.064677 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.053805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.22.157.252:44755)
0928 15:51:16.056299 (+ 2494us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:16.056306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:16.056326 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:51:16.056405 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:16.056411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:16.056546 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:16.056701 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:16.057230 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.059235 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.062191 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.063618 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.063842 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.064068 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:16.064137 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 15:51:16.064285 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2249,"thread_start_us":2464,"threads_started":1}
I20260928 15:51:16.066533 30194 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 15:51:16.067394 30223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d59f65974d44666b7524a34baf610b7. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:16.067705 30223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: This master's current role is: LEARNER
W20260928 15:51:16.123886 30161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:16.124177 30161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:16.124333 30161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:16.132040 30161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:16.132169 30161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:16.132349 30161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:16.142905 30161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab
--principal=oryx/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:38803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=34345
--tserver_master_addrs=127.22.157.254:46257,127.22.157.253:45249,127.22.157.252:44755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:16.144375 30161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:16.145987 30161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:16.159864 30243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:16.159976 30245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:16.159972 30242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:16.161716 30161 server_base.cc:1034] running on GCE node
I20260928 15:51:16.162395 30161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:16.163481 30161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:16.164776 30161 hybrid_clock.cc:648] HybridClock initialized: now 1790610676164664 us; error 60 us; skew 500 ppm
I20260928 15:51:16.165133 30161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610676, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:16.168959 30161 init.cc:377] Logged in from keytab as oryx/127.22.157.193@KRBTEST.COM (short username oryx)
I20260928 15:51:16.170523 30161 webserver.cc:460] Webserver started at http://127.22.157.193:34345/ using document root <none> and password file <none>
I20260928 15:51:16.171190 30161 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:16.171308 30161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:16.177019 30161 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:51:16.179703 30252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:16.181391 30161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:16.181612 30161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "d09dac1b78ef4becac2379eaf9bef297"
format_stamp: "Formatted at 2026-09-28 15:51:12 on dist-test-slave-1vw0"
server_key: "0d484104299c2f1c4f65836daf2361a0"
server_key_iv: "e8bbdd2f7549c8820d61844dae6fc3ef"
server_key_version: "encryptionkey@0"
I20260928 15:51:16.182660 30161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:16.212378 30161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:16.217870 30161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:16.218242 30161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:16.219367 30161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:16.220991 30161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:16.221110 30161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:16.221258 30161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:16.221371 30161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:16.258744 30161 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:38803
I20260928 15:51:16.258771 30366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:38803 every 8 connection(s)
I20260928 15:51:16.260646 30161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:16.265146 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 30161
I20260928 15:51:16.265581 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 29585
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610676, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.252@KRBTEST.COM
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610676, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.253@KRBTEST.COM
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.194@KRBTEST.COM
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610676, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:16.309330 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.264129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:56417 (local address 127.22.157.253:45249)
0928 15:51:16.284913 (+ 20784us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:16.284924 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:16.284965 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:51:16.285059 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:16.285068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:16.285204 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:16.285346 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:16.289589 (+ 4243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.290760 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.293813 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.294523 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.294626 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.301441 (+ 6815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:16.301448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:16.301453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:16.301495 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:16.305217 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.306228 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.306234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.306238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.306424 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.306927 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.306932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.306936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.307323 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:16.307471 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:16.308551 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0928 15:51:16.308851 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20599,"thread_start_us":20430,"threads_started":1}
I20260928 15:51:16.309930 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.282336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:55745 (local address 127.22.157.252:44755)
0928 15:51:16.282602 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:16.282612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:16.286291 (+ 3679us) server_negotiation.cc:407] Connection header received
0928 15:51:16.286518 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:16.286523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:16.286601 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:16.286708 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:16.289591 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.290758 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.293811 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.294527 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.294626 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.301495 (+ 6869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:16.301503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:16.301508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:16.301554 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:16.306123 (+ 4569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.307225 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.307231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.307235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.307416 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.307975 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.307982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.307986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.308400 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:16.308571 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:16.309206 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 15:51:16.309514 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:51:16.311930 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.282833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44755 (local address 127.22.157.193:55745)
0928 15:51:16.286187 (+ 3354us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:16.286206 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:16.286382 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:16.286731 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:16.286735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:16.287901 (+ 1166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:16.289351 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.289381 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.290986 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.290992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.293639 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.293654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.295030 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.295039 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.295500 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.296503 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:16.296555 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:16.301163 (+ 4608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:16.306309 (+ 5146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.306315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.306319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.307043 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.307596 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.307602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.307613 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.307819 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.308669 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:16.308675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:16.308877 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 15:51:16.310130 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0928 15:51:16.310477 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3239,"mutex_wait_us":449,"spinlock_wait_cycles":4608,"thread_start_us":3117,"threads_started":1}
I20260928 15:51:16.311951 30373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.265043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.22.157.193:56417)
0928 15:51:16.276377 (+ 11334us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:16.276474 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:16.280150 (+ 3676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:16.286615 (+ 6465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:16.286634 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:16.287901 (+ 1267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:16.289351 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.289381 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.290986 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.290992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.293639 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.293654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.295030 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.295039 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.295495 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.296570 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:16.296587 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:16.301163 (+ 4576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:16.305435 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.305448 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.305473 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.306093 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.306576 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.306580 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.306584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.306804 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.307548 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:16.307557 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:16.308087 (+ 530us) client_negotiation.cc:769] Sending connection context
0928 15:51:16.309640 (+ 1553us) client_negotiation.cc:240] Negotiation successful
0928 15:51:16.310262 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10833,"mutex_wait_us":485,"spinlock_wait_cycles":14848,"thread_start_us":15568,"threads_started":1}
I20260928 15:51:16.313997 30369 heartbeater.cc:347] Connected to a master server at 127.22.157.253:45249
I20260928 15:51:16.314154 30367 heartbeater.cc:347] Connected to a master server at 127.22.157.252:44755
I20260928 15:51:16.314407 30369 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:16.314415 30367 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:16.315513 30369 heartbeater.cc:511] Master 127.22.157.253:45249 requested a full tablet report, sending...
I20260928 15:51:16.315543 30367 heartbeater.cc:511] Master 127.22.157.252:44755 requested a full tablet report, sending...
I20260928 15:51:16.316953 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.265055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.193:56767)
0928 15:51:16.285941 (+ 20886us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:16.285970 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:16.286135 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:16.298605 (+ 12470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:16.298612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:16.299078 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:16.299370 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.299386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.300821 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.300829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.302435 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.302447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.303471 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.303492 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.303713 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.304229 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:16.304252 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:16.307845 (+ 3593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:16.313051 (+ 5206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.313058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.313063 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.313655 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.314194 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.314199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.314203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.314431 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.315191 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:16.315200 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:16.315864 (+ 664us) client_negotiation.cc:769] Sending connection context
0928 15:51:16.316023 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 15:51:16.316368 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20705,"mutex_wait_us":41,"spinlock_wait_cycles":71040,"thread_start_us":20359,"threads_started":1}
I20260928 15:51:16.317039 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.264161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:56767 (local address 127.22.157.254:46257)
0928 15:51:16.298204 (+ 34043us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:16.298208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:16.298226 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:51:16.298279 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:16.298283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:16.298401 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:16.298537 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:16.299565 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.300628 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.302635 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.303255 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.303992 (+ 737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.308074 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:16.308080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:16.308085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:16.308127 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:16.312869 (+ 4742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.313835 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.313844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.313852 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.314018 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.314607 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.314613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.314616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.315022 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:16.316115 (+ 1093us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:16.316296 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 15:51:16.316557 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33917,"thread_start_us":33798,"threads_started":1}
I20260928 15:51:16.318321 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.266801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.22.157.194:32891)
0928 15:51:16.285353 (+ 18552us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:16.285367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:16.287853 (+ 2486us) server_negotiation.cc:407] Connection header received
0928 15:51:16.288257 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:16.288273 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:16.288707 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:16.289010 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:16.292435 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.294869 (+ 2434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.298892 (+ 4023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.300003 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.300129 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.308440 (+ 8311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:16.308461 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:16.308485 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:16.308542 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:16.313786 (+ 5244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.315289 (+ 1503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.315300 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.315303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.315509 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.316042 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.316048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.316051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.316486 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:16.316907 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:16.317662 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 15:51:16.317940 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18329,"thread_start_us":18136,"threads_started":1}
I20260928 15:51:16.319039 30184 ts_manager.cc:194] Registered new tserver with Master: d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803)
I20260928 15:51:16.319558 29709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:16.320394 30368 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46257
I20260928 15:51:16.320565 30368 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:16.321194 30368 heartbeater.cc:511] Master 127.22.157.254:46257 requested a full tablet report, sending...
I20260928 15:51:16.321686 29995 ts_manager.cc:194] Registered new tserver with Master: d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803)
I20260928 15:51:16.322445 29388 ts_manager.cc:194] Re-registered known tserver with Master: d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803)
I20260928 15:51:16.323355 29388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:56767
I20260928 15:51:16.408195 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 29585
I20260928 15:51:16.425315 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:32891
--local_ip_for_outbound_sockets=127.22.157.194
--tserver_master_addrs=127.22.157.254:46257
--webserver_port=45755
--webserver_interface=127.22.157.194
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab
--principal=oryx/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.157.254:46257,127.22.157.253:45249
--tserver_master_addrs=127.22.157.254:46257,127.22.157.253:45249,127.22.157.252:44755 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:51:16.436611 30143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: NON_VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: true } } }
I20260928 15:51:16.437264 30143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:16.438728 30380 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: attempting to promote NON_VOTER 380d381cd4b44b80bafd3a5fc51beb24 to VOTER
I20260928 15:51:16.439296 30380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } }
I20260928 15:51:16.441227 30194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 LEARNER]: Refusing update from remote peer 4d59f65974d44666b7524a34baf610b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 15:51:16.441385 30005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Refusing update from remote peer 4d59f65974d44666b7524a34baf610b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 15:51:16.441838 30381 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 15:51:16.442457 30143 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 15:51:16.445683 30143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:16.446208 30005 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:16.451488 30381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:16.451918 30381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:16.452153 30143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:16.452445 30143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:16.459069 30384 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:16.459504 30384 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:51:16.460460 30381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:16.460729 30381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:16.461259 30194 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:16.464464 30223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d59f65974d44666b7524a34baf610b7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:16.464749 30223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 15:51:16.779758 30379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:16.780105 30379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:16.780285 30379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:16.788229 30379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:16.788410 30379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:16.788605 30379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:51:16.799551 30379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab
--principal=oryx/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:32891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=45755
--tserver_master_addrs=127.22.157.254:46257,127.22.157.253:45249,127.22.157.252:44755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:16.801260 30379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:16.802842 30379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:16.819478 30389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:16.819449 30390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:16.819543 30392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:16.820019 30379 server_base.cc:1034] running on GCE node
I20260928 15:51:16.820986 30379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:16.822165 30379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:16.823416 30379 hybrid_clock.cc:648] HybridClock initialized: now 1790610676823372 us; error 41 us; skew 500 ppm
I20260928 15:51:16.823823 30379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610676, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:16.827721 30379 init.cc:377] Logged in from keytab as oryx/127.22.157.194@KRBTEST.COM (short username oryx)
I20260928 15:51:16.829694 30379 webserver.cc:460] Webserver started at http://127.22.157.194:45755/ using document root <none> and password file <none>
I20260928 15:51:16.830487 30379 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:16.830665 30379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:16.837531 30379 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 15:51:16.840988 30399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:16.842832 30379 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:51:16.843055 30379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "0bda64e44c8f4422ad664c137df59822"
format_stamp: "Formatted at 2026-09-28 15:51:13 on dist-test-slave-1vw0"
server_key: "2a76056a7ddd0f308413db5e1e2ee11c"
server_key_iv: "8df70c1131d672f396dcceb8089141b6"
server_key_version: "encryptionkey@0"
I20260928 15:51:16.844143 30379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:16.882047 30379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:16.886931 30379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:16.887252 30379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:16.888347 30379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:16.890013 30379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:16.890141 30379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:16.890290 30379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:16.890378 30379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:16.926970 30379 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:32891
I20260928 15:51:16.927070 30514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:32891 every 8 connection(s)
I20260928 15:51:16.928720 30379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:51:16.931079 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 30379
I20260928 15:51:16.931406 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 29720
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.195@KRBTEST.COM
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610676, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.252@KRBTEST.COM
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610676, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.253@KRBTEST.COM
Sep 28 15:51:16 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610676, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:16.975251 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.945566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:42383 (local address 127.22.157.254:46257)
0928 15:51:16.946079 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:16.946088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:16.946981 (+ 893us) server_negotiation.cc:407] Connection header received
0928 15:51:16.953670 (+ 6689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:16.953678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:16.953781 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:16.953868 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:16.956403 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.957578 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.961042 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.961738 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.961857 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.967592 (+ 5735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:16.967601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:16.967608 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:16.967671 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:16.971285 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.972342 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.972349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.972352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.972527 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.973087 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.973093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.973097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.973495 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:16.973799 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:16.974420 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 15:51:16.974818 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260928 15:51:16.975260 30519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.931988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:44579 (local address 127.22.157.252:44755)
0928 15:51:16.941247 (+ 9259us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:16.941252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:16.944457 (+ 3205us) server_negotiation.cc:407] Connection header received
0928 15:51:16.950609 (+ 6152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:16.950621 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:16.950760 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:16.950872 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:16.954362 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.955602 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.959394 (+ 3792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.960217 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.960342 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.966227 (+ 5885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:16.966235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:16.966241 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:16.966289 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:16.970078 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.971242 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.971249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.971252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.971439 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.972154 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.972160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.972163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.972589 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:16.972761 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:16.974270 (+ 1509us) server_negotiation.cc:299] Negotiation successful
0928 15:51:16.974818 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9127,"thread_start_us":8949,"threads_started":1}
I20260928 15:51:16.978654 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.946104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:42383)
0928 15:51:16.946882 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:16.946903 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:16.953340 (+ 6437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:16.954141 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:16.954146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:16.955752 (+ 1606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:16.956233 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.956246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.958590 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.958596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.960913 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.960928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.962315 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.962323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.962506 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.963174 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:16.963195 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:16.967357 (+ 4162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:16.971459 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.971464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.971467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.972186 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.972688 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.972693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.972696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.972921 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.973890 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:16.973894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:16.974086 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 15:51:16.975529 (+ 1443us) client_negotiation.cc:240] Negotiation successful
0928 15:51:16.976716 (+ 1187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":507,"thread_start_us":443,"threads_started":1}
I20260928 15:51:16.978675 30522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.932689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44755 (local address 127.22.157.194:44579)
0928 15:51:16.944194 (+ 11505us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:16.944270 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:16.950075 (+ 5805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:16.951760 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:16.951777 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:16.952961 (+ 1184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:16.954065 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.954132 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.955834 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.955841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.959175 (+ 3334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.959193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.960986 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.960996 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.961553 (+ 557us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.962391 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:16.962465 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:16.965884 (+ 3419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:16.970316 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.970329 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.970351 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.971072 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.971671 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.971675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.971678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.971997 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.972877 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:16.972887 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:16.973679 (+ 792us) client_negotiation.cc:769] Sending connection context
0928 15:51:16.975637 (+ 1958us) client_negotiation.cc:240] Negotiation successful
0928 15:51:16.976598 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10969,"mutex_wait_us":772,"spinlock_wait_cycles":20352,"thread_start_us":13041,"threads_started":1}
I20260928 15:51:16.979354 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.932665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.22.157.195:45159)
0928 15:51:16.945509 (+ 12844us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:16.945516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:16.946313 (+ 797us) server_negotiation.cc:407] Connection header received
0928 15:51:16.946419 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:16.946432 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:16.946815 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:16.947006 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:16.950422 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.952655 (+ 2233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.955015 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.956056 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.956196 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.964023 (+ 7827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:16.964051 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:16.964075 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:16.964132 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:16.970993 (+ 6861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.974237 (+ 3244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.974250 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.974253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.974484 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.976133 (+ 1649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.976142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.976148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.976898 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:16.977346 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:16.978304 (+ 958us) server_negotiation.cc:299] Negotiation successful
0928 15:51:16.978716 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12654,"thread_start_us":12475,"threads_started":1}
I20260928 15:51:16.980929 30516 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46257
I20260928 15:51:16.981323 30516 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:16.981453 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.947250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.22.157.194:45419)
0928 15:51:16.954235 (+ 6985us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:16.954257 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:16.954608 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:16.955501 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:16.955506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:16.956286 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:16.957426 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.957442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.958533 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.958550 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.960848 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:16.960867 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.962126 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.962134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:16.962339 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.962960 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:16.962990 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:16.966002 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:16.971208 (+ 5206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.971213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.971217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.971908 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.974685 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:16.974693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:16.974700 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:16.974931 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:16.980082 (+ 5151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:16.980088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:16.980322 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 15:51:16.980545 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 15:51:16.981003 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6797,"mutex_wait_us":140,"thread_start_us":6542,"threads_started":1}
I20260928 15:51:16.982385 30516 heartbeater.cc:511] Master 127.22.157.254:46257 requested a full tablet report, sending...
I20260928 15:51:16.980962 30515 heartbeater.cc:347] Connected to a master server at 127.22.157.252:44755
I20260928 15:51:16.983347 30515 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:16.983552 29843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:16.983955 30515 heartbeater.cc:511] Master 127.22.157.252:44755 requested a full tablet report, sending...
I20260928 15:51:16.984687 29388 ts_manager.cc:194] Re-registered known tserver with Master: 0bda64e44c8f4422ad664c137df59822 (127.22.157.194:32891)
I20260928 15:51:16.985340 30526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:16.947400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:45419 (local address 127.22.157.253:45249)
0928 15:51:16.954733 (+ 7333us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:16.954741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:16.954766 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:51:16.955172 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:16.955178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:16.955261 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:16.955366 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:16.957633 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.958383 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.961088 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:16.961735 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:16.961855 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:16.966256 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:16.966261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:16.966265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:16.966314 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:16.969990 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.973045 (+ 3055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.973052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.973055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.973261 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:16.979458 (+ 6197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:16.979465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:16.979469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:16.979928 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:16.980107 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:16.984631 (+ 4524us) server_negotiation.cc:299] Negotiation successful
0928 15:51:16.984942 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7210,"thread_start_us":7210,"threads_started":1}
I20260928 15:51:16.985488 30184 ts_manager.cc:194] Registered new tserver with Master: 0bda64e44c8f4422ad664c137df59822 (127.22.157.194:32891)
I20260928 15:51:16.986429 29388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:42383
I20260928 15:51:16.986493 30517 heartbeater.cc:347] Connected to a master server at 127.22.157.253:45249
I20260928 15:51:16.986644 30517 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:16.987179 30517 heartbeater.cc:511] Master 127.22.157.253:45249 requested a full tablet report, sending...
I20260928 15:51:16.988677 29995 ts_manager.cc:194] Registered new tserver with Master: 0bda64e44c8f4422ad664c137df59822 (127.22.157.194:32891)
I20260928 15:51:17.078868 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 29720
I20260928 15:51:17.095973 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:45159
--local_ip_for_outbound_sockets=127.22.157.195
--tserver_master_addrs=127.22.157.254:46257
--webserver_port=41951
--webserver_interface=127.22.157.195
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab
--principal=oryx/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.157.254:46257,127.22.157.253:45249
--tserver_master_addrs=127.22.157.254:46257,127.22.157.253:45249,127.22.157.252:44755 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:51:17.326663 30368 heartbeater.cc:503] Master 127.22.157.254:46257 was elected leader, sending a full tablet report...
W20260928 15:51:17.443719 30530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:17.444096 30530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:17.444265 30530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:17.452881 30530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:17.453037 30530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:17.453224 30530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:51:17.463812 30530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab
--principal=oryx/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:45159
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=41951
--tserver_master_addrs=127.22.157.254:46257,127.22.157.253:45249,127.22.157.252:44755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:17.465332 30530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:17.466717 30530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:17.480598 30535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:17.480589 30536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:17.480851 30538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:17.481472 30530 server_base.cc:1034] running on GCE node
I20260928 15:51:17.482146 30530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:17.483147 30530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:17.484396 30530 hybrid_clock.cc:648] HybridClock initialized: now 1790610677484329 us; error 63 us; skew 500 ppm
I20260928 15:51:17.484807 30530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:17 dist-test-slave-1vw0 krb5kdc[29343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610677, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:17.488256 30530 init.cc:377] Logged in from keytab as oryx/127.22.157.195@KRBTEST.COM (short username oryx)
I20260928 15:51:17.489902 30530 webserver.cc:460] Webserver started at http://127.22.157.195:41951/ using document root <none> and password file <none>
I20260928 15:51:17.490613 30530 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:17.490757 30530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:17.496130 30530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 15:51:17.498827 30545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:17.500276 30530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:51:17.500464 30530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2"
format_stamp: "Formatted at 2026-09-28 15:51:14 on dist-test-slave-1vw0"
server_key: "dec6f44d02bacb7490d0202336d6866e"
server_key_iv: "891df3208080f5f0e6dbf1139cad5af7"
server_key_version: "encryptionkey@0"
I20260928 15:51:17.501703 30530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:17.526356 30530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:17.531255 30530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:17.531567 30530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:17.532598 30530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:17.534026 30530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:17.534128 30530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:17.534263 30530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:17.534360 30530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:17.568446 30530 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:45159
I20260928 15:51:17.568516 30657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:45159 every 8 connection(s)
I20260928 15:51:17.569988 30530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:51:17.571143 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 30530
Sep 28 15:51:17 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610677, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
Sep 28 15:51:17 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610677, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.253@KRBTEST.COM
Sep 28 15:51:17 dist-test-slave-1vw0 krb5kdc[29343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610677, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.252@KRBTEST.COM
I20260928 15:51:17.614573 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.573097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:59683 (local address 127.22.157.254:46257)
0928 15:51:17.590276 (+ 17179us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:17.590285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:17.590314 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:51:17.590395 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:17.590403 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:17.590509 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:17.592922 (+ 2413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:17.593620 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.594338 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.597382 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.597818 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.597968 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.603711 (+ 5743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:17.603720 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:17.603727 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:17.603779 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:17.608575 (+ 4796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.610565 (+ 1990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.610572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.610576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.610738 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.611569 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.611575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.611579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.611960 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:17.612119 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:17.613880 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0928 15:51:17.614156 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17004,"thread_start_us":17552,"threads_started":1}
I20260928 15:51:17.616317 30667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.574693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:50761 (local address 127.22.157.253:45249)
0928 15:51:17.591751 (+ 17058us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:17.591758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:17.591781 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:51:17.591844 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:17.591850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:17.591936 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:17.592055 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:17.594569 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.595683 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.598228 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.598877 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.599014 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.603929 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:17.603936 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:17.603941 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:17.603985 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:17.610371 (+ 6386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.612385 (+ 2014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.612390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.612394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.612632 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.613138 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.613146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.613150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.613538 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:17.613700 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:17.615681 (+ 1981us) server_negotiation.cc:299] Negotiation successful
0928 15:51:17.615922 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16921,"thread_start_us":17458,"threads_started":1}
I20260928 15:51:17.616575 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.573958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.22.157.195:50761)
0928 15:51:17.581796 (+ 7838us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:17.581867 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:17.583676 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:17.592116 (+ 8440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:17.592124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:17.592669 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:17.593387 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.593414 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.595848 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.595854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.597251 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.597261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.599079 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.599087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.599292 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.599892 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:17.599911 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:17.603695 (+ 3784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:17.610910 (+ 7215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:17.610916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:17.610923 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:17.612231 (+ 1308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:17.612795 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:17.612801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:17.612806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:17.612997 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:17.614986 (+ 1989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:17.614992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:17.615190 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 15:51:17.615337 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 15:51:17.616129 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7485,"mutex_wait_us":233,"thread_start_us":9913,"threads_started":1}
I20260928 15:51:17.617419 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.573994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.195:59683)
0928 15:51:17.584349 (+ 10355us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:17.584367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:17.584491 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:17.591195 (+ 6704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:17.591214 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:17.592318 (+ 1104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:17.593387 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.593414 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.594526 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.594533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.596808 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.596825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.598062 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.598070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.598363 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.599696 (+ 1333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:17.599766 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:17.603368 (+ 3602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:17.608768 (+ 5400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:17.608785 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:17.608813 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:17.609800 (+ 987us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:17.610867 (+ 1067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:17.610876 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:17.610883 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:17.611298 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:17.612096 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:17.612110 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:17.612849 (+ 739us) client_negotiation.cc:769] Sending connection context
0928 15:51:17.613296 (+ 447us) client_negotiation.cc:240] Negotiation successful
0928 15:51:17.614013 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10205,"mutex_wait_us":68,"spinlock_wait_cycles":48000,"thread_start_us":7134,"threads_started":1}
I20260928 15:51:17.618419 30660 heartbeater.cc:347] Connected to a master server at 127.22.157.253:45249
I20260928 15:51:17.618850 30660 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:17.620086 30660 heartbeater.cc:511] Master 127.22.157.253:45249 requested a full tablet report, sending...
I20260928 15:51:17.620146 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.584822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:37877 (local address 127.22.157.252:44755)
0928 15:51:17.599507 (+ 14685us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:17.599515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:17.599537 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:51:17.599616 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:17.599622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:17.599720 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:17.599894 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:17.602265 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.603467 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.605931 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.606678 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.606790 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.611303 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:17.611310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:17.611315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:17.611358 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:17.615278 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.617754 (+ 2476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.617759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.617763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.617923 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.618394 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.618402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.618405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.618790 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:17.618931 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:17.619517 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 15:51:17.619769 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14541,"thread_start_us":14443,"threads_started":1}
I20260928 15:51:17.621270 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.584613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44755 (local address 127.22.157.195:37877)
0928 15:51:17.585798 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:17.586301 (+ 503us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:17.586454 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:17.599950 (+ 13496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:17.599963 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:17.601827 (+ 1864us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:17.602093 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.602102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.604048 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.604054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.605659 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.605675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.607118 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.607127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.607313 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.607909 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:17.607934 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:17.611068 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:17.615433 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:17.615437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:17.615441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:17.615909 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:17.618080 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:17.618085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:17.618088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:17.618260 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:17.619026 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:17.619030 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:17.619208 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 15:51:17.620429 (+ 1221us) client_negotiation.cc:240] Negotiation successful
0928 15:51:17.620781 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"thread_start_us":909,"threads_started":1}
I20260928 15:51:17.622871 29995 ts_manager.cc:194] Registered new tserver with Master: d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159)
I20260928 15:51:17.623616 30659 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46257
I20260928 15:51:17.623775 30659 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:17.624354 30659 heartbeater.cc:511] Master 127.22.157.254:46257 requested a full tablet report, sending...
I20260928 15:51:17.625661 29388 ts_manager.cc:194] Re-registered known tserver with Master: d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159)
I20260928 15:51:17.627054 29388 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:59683
I20260928 15:51:17.633930 30658 heartbeater.cc:347] Connected to a master server at 127.22.157.252:44755
I20260928 15:51:17.634151 30658 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:17.634694 30658 heartbeater.cc:511] Master 127.22.157.252:44755 requested a full tablet report, sending...
I20260928 15:51:17.636286 30184 ts_manager.cc:194] Registered new tserver with Master: d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159)
I20260928 15:51:17.644378 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.617517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.22.157.252:44755)
0928 15:51:17.620378 (+ 2861us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:17.620385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:17.623477 (+ 3092us) server_negotiation.cc:407] Connection header received
0928 15:51:17.623691 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:17.623697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:17.623789 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:17.623882 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:17.626247 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.627403 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.632016 (+ 4613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.632827 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.632966 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.637100 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:17.637105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:17.637109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:17.637143 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:17.640606 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.641509 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.641515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.641518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.641684 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.642206 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.642211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.642214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.642546 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:17.642837 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:17.643731 (+ 894us) server_negotiation.cc:299] Negotiation successful
0928 15:51:17.643981 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2746,"thread_start_us":14936,"threads_started":1}
I20260928 15:51:17.644650 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.606937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.22.157.254:46257)
0928 15:51:17.614917 (+ 7980us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:17.614923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:17.619270 (+ 4347us) server_negotiation.cc:407] Connection header received
0928 15:51:17.619553 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:17.619559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:17.619632 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:17.619703 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:17.623731 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.624907 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.630466 (+ 5559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.631420 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.631624 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.638350 (+ 6726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:17.638356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:17.638359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:17.638396 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:17.641638 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.642593 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.642599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.642602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.642732 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.643151 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.643156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.643159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.643444 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:17.643715 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:17.644037 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 15:51:17.644271 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7858,"thread_start_us":16724,"threads_started":1}
I20260928 15:51:17.646023 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.604839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.22.157.253:45249)
0928 15:51:17.623915 (+ 19076us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:17.623922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:17.623944 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:51:17.624003 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:17.624008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:17.624094 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:17.624216 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:17.627430 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.628784 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.631056 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.631820 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.631976 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.638355 (+ 6379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:17.638363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:17.638367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:17.638410 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:17.641764 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.642928 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.642936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.642941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.643134 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:17.643744 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:17.643749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:17.643752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:17.644110 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:17.644324 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:17.645425 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0928 15:51:17.645642 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1719,"thread_start_us":1569,"threads_started":1}
I20260928 15:51:17.649166 29388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:51:17.656997 29388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:51:17.693651 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.679465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.22.157.193:38803)
0928 15:51:17.684199 (+ 4734us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:17.684205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:17.685327 (+ 1122us) server_negotiation.cc:407] Connection header received
0928 15:51:17.686283 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:17.686291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:17.686559 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:17.686769 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:17.687720 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.689187 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.691376 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.692566 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.692763 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.692947 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:17.693113 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 15:51:17.693288 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4534,"thread_start_us":4430,"threads_started":1}
I20260928 15:51:17.693637 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.679195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:38803 (local address 127.0.0.1:39876)
0928 15:51:17.685225 (+ 6030us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:17.685241 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:17.686124 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:17.686967 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:17.686972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:17.687044 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:17.687513 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.687528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.689334 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.689337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.691230 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.691242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.692795 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.692801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.692914 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.692938 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:51:17.693060 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 15:51:17.693197 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5903,"spinlock_wait_cycles":3968,"thread_start_us":5119,"threads_started":1}
I20260928 15:51:17.693979 30693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.678768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:45159 (local address 127.0.0.1:37480)
0928 15:51:17.684587 (+ 5819us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:17.684605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:17.684757 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:17.685419 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:17.685423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:17.685441 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:17.685941 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.685953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.688129 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.688141 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.690289 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.690301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.693105 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.693112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.693277 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.693310 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:51:17.693402 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:51:17.693545 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5679,"mutex_wait_us":21,"thread_start_us":6346,"threads_started":1}
I20260928 15:51:17.695150 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.679035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.22.157.195:45159)
0928 15:51:17.680028 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:17.680037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:17.684716 (+ 4679us) server_negotiation.cc:407] Connection header received
0928 15:51:17.684927 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:17.684933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:17.685215 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:17.685457 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:17.686194 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.687985 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.690454 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.691737 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.691902 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.692090 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:17.694674 (+ 2584us) server_negotiation.cc:299] Negotiation successful
0928 15:51:17.694809 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":789,"threads_started":1}
I20260928 15:51:17.695346 30696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.684673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:32891 (local address 127.0.0.1:53376)
0928 15:51:17.686032 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:17.686082 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:17.687018 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:17.687742 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:17.687746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:17.687772 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:17.688190 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.688203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.690087 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.690093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.692569 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:17.692582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.694582 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.694588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:17.694752 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.694782 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:51:17.694872 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 15:51:17.695003 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"mutex_wait_us":3,"spinlock_wait_cycles":1408,"thread_start_us":1113,"threads_started":1}
I20260928 15:51:17.695561 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:17.684960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.22.157.194:32891)
0928 15:51:17.685583 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:17.685592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:17.686164 (+ 572us) server_negotiation.cc:407] Connection header received
0928 15:51:17.687250 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:17.687257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:17.687556 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:17.687732 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:17.688395 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.689904 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.692770 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:17.694274 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:17.694464 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:17.694590 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:17.694933 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 15:51:17.695107 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":372,"threads_started":1}
I20260928 15:51:17.698010 30301 tablet_service.cc:1467] Processing CreateTablet for tablet cda2b7d091b840fea5d1729f2c32bc59 (DEFAULT_TABLE table=test-table [id=2c2636d9b4a6463aace9b1ac5f5eabc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:17.699261 30301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda2b7d091b840fea5d1729f2c32bc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:17.698727 30593 tablet_service.cc:1467] Processing CreateTablet for tablet cda2b7d091b840fea5d1729f2c32bc59 (DEFAULT_TABLE table=test-table [id=2c2636d9b4a6463aace9b1ac5f5eabc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:17.699005 30448 tablet_service.cc:1467] Processing CreateTablet for tablet cda2b7d091b840fea5d1729f2c32bc59 (DEFAULT_TABLE table=test-table [id=2c2636d9b4a6463aace9b1ac5f5eabc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:17.700155 30593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda2b7d091b840fea5d1729f2c32bc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:17.700150 30448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda2b7d091b840fea5d1729f2c32bc59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:17.712630 30697 tablet_bootstrap.cc:492] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: Bootstrap starting.
I20260928 15:51:17.717001 30697 tablet_bootstrap.cc:654] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:17.717501 30699 tablet_bootstrap.cc:492] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: Bootstrap starting.
I20260928 15:51:17.719385 30697 log.cc:824] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:17.720495 30698 tablet_bootstrap.cc:492] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: Bootstrap starting.
I20260928 15:51:17.721388 30699 tablet_bootstrap.cc:654] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:17.722404 30697 tablet_bootstrap.cc:492] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: No bootstrap required, opened a new log
I20260928 15:51:17.722684 30697 ts_tablet_manager.cc:1397] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260928 15:51:17.722714 30699 log.cc:824] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:17.724045 30698 tablet_bootstrap.cc:654] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:17.725461 30698 log.cc:824] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:17.726212 30699 tablet_bootstrap.cc:492] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: No bootstrap required, opened a new log
I20260928 15:51:17.726519 30699 ts_tablet_manager.cc:1397] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 15:51:17.728549 30698 tablet_bootstrap.cc:492] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: No bootstrap required, opened a new log
I20260928 15:51:17.728883 30698 ts_tablet_manager.cc:1397] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 15:51:17.733692 30697 raft_consensus.cc:348] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:17.734215 30697 raft_consensus.cc:374] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:17.734326 30697 raft_consensus.cc:729] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09dac1b78ef4becac2379eaf9bef297, State: Initialized, Role: FOLLOWER
I20260928 15:51:17.735016 30697 consensus_queue.cc:260] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:17.737339 30698 raft_consensus.cc:348] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:17.737828 30698 raft_consensus.cc:374] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:17.737926 30698 raft_consensus.cc:729] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bda64e44c8f4422ad664c137df59822, State: Initialized, Role: FOLLOWER
I20260928 15:51:17.737939 30699 raft_consensus.cc:348] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:17.738284 30697 ts_tablet_manager.cc:1428] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260928 15:51:17.738396 30699 raft_consensus.cc:374] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:17.738503 30699 raft_consensus.cc:729] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b3c169d6b8401eb1d32fa23e69d8c2, State: Initialized, Role: FOLLOWER
I20260928 15:51:17.738660 30698 consensus_queue.cc:260] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:17.739480 30699 consensus_queue.cc:260] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:17.746764 30659 heartbeater.cc:503] Master 127.22.157.254:46257 was elected leader, sending a full tablet report...
I20260928 15:51:17.747445 30698 ts_tablet_manager.cc:1428] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
I20260928 15:51:17.748939 30516 heartbeater.cc:503] Master 127.22.157.254:46257 was elected leader, sending a full tablet report...
I20260928 15:51:17.751114 30699 ts_tablet_manager.cc:1428] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: Time spent starting tablet: real 0.024s user 0.014s sys 0.008s
W20260928 15:51:17.764629 30370 tablet.cc:2369] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:51:17.837230 30661 tablet.cc:2369] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:51:17.937834 30518 tablet.cc:2369] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:51:18.015005 30704 raft_consensus.cc:482] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:18.015362 30704 raft_consensus.cc:504] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:18.017362 30704 leader_election.cc:288] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159), d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803)
I20260928 15:51:18.025128 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.018043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:57569 (local address 127.22.157.193:38803)
0928 15:51:18.018238 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:18.018245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:18.019095 (+ 850us) server_negotiation.cc:407] Connection header received
0928 15:51:18.019428 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:18.019437 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:18.019524 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:18.019574 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:18.020579 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.021420 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.023298 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.024088 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.024273 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.024349 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:18.024693 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 15:51:18.024864 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:51:18.025333 30706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.017845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:38803 (local address 127.22.157.194:57569)
0928 15:51:18.018952 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:18.018976 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:18.019234 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:18.019757 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:18.019760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:18.019793 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:18.020417 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:18.020425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.021558 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.021565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:18.023125 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:18.023140 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.024300 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.024306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:18.024527 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.024553 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:51:18.024657 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 15:51:18.024817 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":107,"spinlock_wait_cycles":1792,"thread_start_us":582,"threads_started":1}
I20260928 15:51:18.025355 30707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.017850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:45159 (local address 127.22.157.194:60057)
0928 15:51:18.019223 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:18.019238 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:18.019367 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:18.019745 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:18.019749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:18.019787 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:18.020390 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:18.020415 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.021530 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.021536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:18.023139 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:18.023153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.024433 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.024440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:18.024612 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.024635 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:51:18.024771 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:51:18.024910 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"mutex_wait_us":721,"spinlock_wait_cycles":768,"thread_start_us":518,"threads_started":1}
I20260928 15:51:18.025183 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.018188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:60057 (local address 127.22.157.195:45159)
0928 15:51:18.018394 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:18.018400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:18.019329 (+ 929us) server_negotiation.cc:407] Connection header received
0928 15:51:18.019505 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:18.019511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:18.019600 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:18.019708 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:18.020607 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.021389 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.023322 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.024203 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.024390 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.024474 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:18.024790 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 15:51:18.024926 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:51:18.026510 30321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda2b7d091b840fea5d1729f2c32bc59" candidate_uuid: "0bda64e44c8f4422ad664c137df59822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09dac1b78ef4becac2379eaf9bef297" is_pre_election: true
I20260928 15:51:18.026846 30705 raft_consensus.cc:482] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:18.026708 30613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda2b7d091b840fea5d1729f2c32bc59" candidate_uuid: "0bda64e44c8f4422ad664c137df59822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" is_pre_election: true
I20260928 15:51:18.027138 30321 raft_consensus.cc:2454] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bda64e44c8f4422ad664c137df59822 in term 0.
I20260928 15:51:18.027149 30705 raft_consensus.cc:504] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:18.027875 30400 leader_election.cc:302] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bda64e44c8f4422ad664c137df59822, d09dac1b78ef4becac2379eaf9bef297; no voters:
I20260928 15:51:18.027941 30613 raft_consensus.cc:2454] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bda64e44c8f4422ad664c137df59822 in term 0.
I20260928 15:51:18.028249 30704 raft_consensus.cc:2781] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:51:18.028447 30704 raft_consensus.cc:482] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:18.028578 30704 raft_consensus.cc:3037] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:18.029103 30705 leader_election.cc:288] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803), 0bda64e44c8f4422ad664c137df59822 (127.22.157.194:32891)
I20260928 15:51:18.032047 30704 raft_consensus.cc:504] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:18.032956 30704 leader_election.cc:288] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [CANDIDATE]: Term 1 election: Requested vote from peers d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159), d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803)
I20260928 15:51:18.033910 30613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda2b7d091b840fea5d1729f2c32bc59" candidate_uuid: "0bda64e44c8f4422ad664c137df59822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2"
I20260928 15:51:18.034153 30613 raft_consensus.cc:3037] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:18.036098 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.030251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:39713 (local address 127.22.157.194:32891)
0928 15:51:18.030444 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:18.030448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:18.030462 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:51:18.030509 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:18.030512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:18.030564 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:18.030668 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:18.031028 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.031724 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.034279 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.035154 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.035275 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.035327 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:18.035719 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 15:51:18.035853 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:51:18.036178 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.029716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:32891 (local address 127.22.157.195:39713)
0928 15:51:18.029892 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:18.029910 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:18.029984 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:18.030680 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:18.030685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:18.030708 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:18.030896 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:18.030904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.031864 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.031869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:18.034022 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:18.034034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.035307 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.035313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:18.035549 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.035583 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:51:18.035689 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:51:18.035837 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":114}
I20260928 15:51:18.036536 30321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda2b7d091b840fea5d1729f2c32bc59" candidate_uuid: "0bda64e44c8f4422ad664c137df59822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09dac1b78ef4becac2379eaf9bef297"
I20260928 15:51:18.036834 30321 raft_consensus.cc:3037] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:18.037091 30468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda2b7d091b840fea5d1729f2c32bc59" candidate_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bda64e44c8f4422ad664c137df59822" is_pre_election: true
I20260928 15:51:18.037570 30468 raft_consensus.cc:2379] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7b3c169d6b8401eb1d32fa23e69d8c2 in current term 1: Already voted for candidate 0bda64e44c8f4422ad664c137df59822 in this term.
I20260928 15:51:18.037875 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.029354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:38803 (local address 127.22.157.195:56321)
0928 15:51:18.029534 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:18.029553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:18.029684 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:18.029997 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:18.030002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:18.030033 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:18.030227 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:18.030238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.031709 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.031714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:18.034033 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:18.034047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.037098 (+ 3051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.037106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:18.037338 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.037383 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 15:51:18.037488 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 15:51:18.037620 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":126}
I20260928 15:51:18.038100 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.029473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:56321 (local address 127.22.157.193:38803)
0928 15:51:18.029644 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:18.029649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:18.029666 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:51:18.029792 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:18.029800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:18.029871 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:18.029961 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:18.030394 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.031526 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.035393 (+ 3867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.036891 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.037058 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.037147 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:18.037534 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 15:51:18.037684 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 15:51:18.039734 30321 raft_consensus.cc:2454] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bda64e44c8f4422ad664c137df59822 in term 1.
I20260928 15:51:18.040127 30400 leader_election.cc:302] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bda64e44c8f4422ad664c137df59822, d09dac1b78ef4becac2379eaf9bef297; no voters:
I20260928 15:51:18.040365 30321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda2b7d091b840fea5d1729f2c32bc59" candidate_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09dac1b78ef4becac2379eaf9bef297" is_pre_election: true
I20260928 15:51:18.040616 30704 raft_consensus.cc:2781] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:18.040758 30321 raft_consensus.cc:2379] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7b3c169d6b8401eb1d32fa23e69d8c2 in current term 1: Already voted for candidate 0bda64e44c8f4422ad664c137df59822 in this term.
I20260928 15:51:18.041196 30704 raft_consensus.cc:686] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 1 LEADER]: Becoming Leader. State: Replica: 0bda64e44c8f4422ad664c137df59822, State: Running, Role: LEADER
I20260928 15:51:18.041388 30613 raft_consensus.cc:2454] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bda64e44c8f4422ad664c137df59822 in term 1.
I20260928 15:51:18.041518 30546 leader_election.cc:302] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7b3c169d6b8401eb1d32fa23e69d8c2; no voters: 0bda64e44c8f4422ad664c137df59822, d09dac1b78ef4becac2379eaf9bef297
I20260928 15:51:18.042002 30705 raft_consensus.cc:2726] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 15:51:18.041923 30704 consensus_queue.cc:237] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } }
I20260928 15:51:18.047715 29388 catalog_manager.cc:5495] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bda64e44c8f4422ad664c137df59822 (127.22.157.194). New cstate: current_term: 1 leader_uuid: "0bda64e44c8f4422ad664c137df59822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } health_report { overall_health: HEALTHY } } }
I20260928 15:51:18.068186 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.062504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.22.157.194:32891)
0928 15:51:18.062723 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:18.062730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:18.062862 (+ 132us) server_negotiation.cc:407] Connection header received
0928 15:51:18.063126 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:18.063130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:18.063208 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:18.063366 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:51:18.063792 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.064599 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.065961 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.066384 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.066538 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.067036 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:51:18.067579 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:51:18.067754 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:18.067804 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 15:51:18.067886 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:51:18.077436 30321 raft_consensus.cc:1261] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 1 FOLLOWER]: Refusing update from remote peer 0bda64e44c8f4422ad664c137df59822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:18.077488 30613 raft_consensus.cc:1261] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 1 FOLLOWER]: Refusing update from remote peer 0bda64e44c8f4422ad664c137df59822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:18.078495 30708 consensus_queue.cc:1035] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:18.078953 30704 consensus_queue.cc:1035] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:18.096248 30713 mvcc.cc:205] Tried to move back new op lower bound from 7334341337394569216 to 7334341337264386048. Current Snapshot: MvccSnapshot[applied={T|T < 7334341337394569216}]
I20260928 15:51:18.116289 29388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:18.117208 29388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:18.124774 29388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d59f65974d44666b7524a34baf610b7: Sending DeleteTablet for 3 replicas of tablet cda2b7d091b840fea5d1729f2c32bc59
I20260928 15:51:18.126492 30301 tablet_service.cc:1514] Processing DeleteTablet for tablet cda2b7d091b840fea5d1729f2c32bc59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:18 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:39876
I20260928 15:51:18.126495 30593 tablet_service.cc:1514] Processing DeleteTablet for tablet cda2b7d091b840fea5d1729f2c32bc59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:18 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:37480
I20260928 15:51:18.126719 30448 tablet_service.cc:1514] Processing DeleteTablet for tablet cda2b7d091b840fea5d1729f2c32bc59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:18 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:53376
I20260928 15:51:18.137535 30724 tablet_replica.cc:330] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: stopping tablet replica
I20260928 15:51:18.138734 30724 raft_consensus.cc:2229] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:51:18.141276 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 29356
I20260928 15:51:18.143038 29418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:18.148095 30722 tablet_replica.cc:330] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: stopping tablet replica
I20260928 15:51:18.148612 30722 raft_consensus.cc:2229] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:18.149153 30722 raft_consensus.cc:2258] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:18.147584 30724 raft_consensus.cc:2258] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:18.151800 30724 ts_tablet_manager.cc:1905] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:18.151793 30722 ts_tablet_manager.cc:1905] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:18.161793 30723 tablet_replica.cc:330] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: stopping tablet replica
I20260928 15:51:18.165454 30722 ts_tablet_manager.cc:1918] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:18.165668 30722 log.cc:1196] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/cda2b7d091b840fea5d1729f2c32bc59
I20260928 15:51:18.166324 30722 ts_tablet_manager.cc:1939] T cda2b7d091b840fea5d1729f2c32bc59 P d7b3c169d6b8401eb1d32fa23e69d8c2: Deleting consensus metadata
I20260928 15:51:18.162362 30723 raft_consensus.cc:2229] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:18.168030 30723 raft_consensus.cc:2258] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:18.172456 30724 ts_tablet_manager.cc:1918] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:18.172684 30724 log.cc:1196] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/cda2b7d091b840fea5d1729f2c32bc59
I20260928 15:51:18.173435 30724 ts_tablet_manager.cc:1939] T cda2b7d091b840fea5d1729f2c32bc59 P 0bda64e44c8f4422ad664c137df59822: Deleting consensus metadata
I20260928 15:51:18.175782 30723 ts_tablet_manager.cc:1905] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:18.179275 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.129182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.22.157.253:45249)
0928 15:51:18.129432 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:18.129442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:18.140195 (+ 10753us) server_negotiation.cc:407] Connection header received
0928 15:51:18.142324 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:18.142332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:18.142434 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:18.142625 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:18.158064 (+ 15439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.159249 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.162444 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.163150 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.163296 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.168542 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:18.168549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:18.168554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:18.168603 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:18.172387 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:18.173834 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:18.173842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:18.173847 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:18.174030 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:18.177288 (+ 3258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:18.177295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:18.177299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:18.177708 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:18.177999 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:18.178625 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 15:51:18.178892 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:51:18.183284 30723 ts_tablet_manager.cc:1918] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:18.183480 30723 log.cc:1196] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/cda2b7d091b840fea5d1729f2c32bc59
I20260928 15:51:18.184062 30723 ts_tablet_manager.cc:1939] T cda2b7d091b840fea5d1729f2c32bc59 P d09dac1b78ef4becac2379eaf9bef297: Deleting consensus metadata
I20260928 15:51:18.197969 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.140372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.22.157.252:44755)
0928 15:51:18.140594 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:18.140601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:18.180307 (+ 39706us) server_negotiation.cc:407] Connection header received
0928 15:51:18.180593 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:18.180601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:18.180693 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:18.180811 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:18.182831 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.183993 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.186044 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.186729 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.186837 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.190990 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:18.190998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:18.191003 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:18.191086 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:18.194762 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:18.195659 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:18.195666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:18.195670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:18.195822 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:18.196286 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:18.196292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:18.196295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:18.196685 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:18.196985 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:18.197372 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 15:51:18.197616 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 15:51:18.300264 29373 catalog_manager.cc:4841] TS d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159): tablet cda2b7d091b840fea5d1729f2c32bc59 (table test-table [id=2c2636d9b4a6463aace9b1ac5f5eabc3]) successfully deleted
I20260928 15:51:18.300448 29372 catalog_manager.cc:4841] TS d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803): tablet cda2b7d091b840fea5d1729f2c32bc59 (table test-table [id=2c2636d9b4a6463aace9b1ac5f5eabc3]) successfully deleted
I20260928 15:51:18.301168 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 29356
I20260928 15:51:18.359248 30674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.338275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.22.157.253:45249)
0928 15:51:18.338560 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:18.338570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:18.339775 (+ 1205us) server_negotiation.cc:407] Connection header received
0928 15:51:18.340324 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:18.340338 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:18.340411 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:18.340532 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:18.343340 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.344529 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.346033 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.346868 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.347029 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.351015 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:18.351023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:18.351029 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:18.351084 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:18.354763 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:18.355824 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:18.355831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:18.355834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:18.355978 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:18.356464 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:18.356469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:18.356472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:18.356960 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:18.357218 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:18.358548 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0928 15:51:18.358827 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 15:51:18.359578 30669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:18.339983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.22.157.252:44755)
0928 15:51:18.340244 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:18.340250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:18.340274 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:51:18.340820 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:18.340826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:18.340912 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:18.341023 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:18.344816 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.346001 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.347811 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:18.348533 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:18.348675 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:18.352152 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:18.352158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:18.352161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:18.352202 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:18.355197 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:18.356165 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:18.356173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:18.356179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:18.356339 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:18.357376 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:18.357381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:18.357388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:18.357877 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:18.358354 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:18.358875 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 15:51:18.359172 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:51:19.103341 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.102391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.193:52495)
0928 15:51:19.103027 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.103153 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":310,"threads_started":1}
W20260928 15:51:19.103767 30368 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:46257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:19.106868 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.106089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.195:51425)
0928 15:51:19.106476 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.106628 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":190,"threads_started":1}
W20260928 15:51:19.107348 30659 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:46257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:19.112494 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.111642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:47811)
0928 15:51:19.112126 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.112260 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":316,"threads_started":1}
W20260928 15:51:19.112912 30516 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:46257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:19.625362 30736 raft_consensus.cc:482] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d59f65974d44666b7524a34baf610b7)
I20260928 15:51:19.625590 30736 raft_consensus.cc:504] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } }
I20260928 15:51:19.626320 30737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d59f65974d44666b7524a34baf610b7)
I20260928 15:51:19.626663 30736 leader_election.cc:288] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257), 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252:44755)
I20260928 15:51:19.626613 30737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } }
I20260928 15:51:19.634672 30737 leader_election.cc:288] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257), e588dbb8beb24471ac35e7ee90a05417 (127.22.157.253:45249)
I20260928 15:51:19.638329 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.626921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51294)
0928 15:51:19.637626 (+ 10705us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.637926 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10500,"threads_started":1}
I20260928 15:51:19.640974 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.629206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51306)
0928 15:51:19.640227 (+ 11021us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.640550 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10822,"threads_started":1}
W20260928 15:51:19.643155 30171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
W20260928 15:51:19.643155 29981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:19.650117 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.649144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51322)
0928 15:51:19.649609 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.649817 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273}
W20260928 15:51:19.650425 30171 leader_election.cc:334] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257): Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:19.650635 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.649877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51336)
0928 15:51:19.650093 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.650345 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:51:19.650830 29981 leader_election.cc:334] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257): Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:19.653782 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.626943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44755 (local address 127.0.0.1:47618)
0928 15:51:19.638662 (+ 11719us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.638693 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:19.638927 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:19.643637 (+ 4710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:19.643656 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:19.643697 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:19.644001 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:19.644013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:19.647992 (+ 3979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:19.647997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:19.651140 (+ 3143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:19.651162 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:19.652944 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:19.652953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:19.653139 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:19.653170 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 15:51:19.653271 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:51:19.653429 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11563,"thread_start_us":10292,"threads_started":1}
I20260928 15:51:19.653875 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.629237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:45249 (local address 127.0.0.1:38326)
0928 15:51:19.641304 (+ 12067us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.641330 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:19.641521 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:19.642352 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:19.642361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:19.642406 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:19.642779 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:19.642792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:19.644644 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:19.644651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:19.651187 (+ 6536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:19.651210 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:19.652917 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:19.652926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:19.653132 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:19.653169 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:51:19.653271 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 15:51:19.653437 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11913,"mutex_wait_us":3294,"thread_start_us":12413,"threads_started":1}
I20260928 15:51:19.653884 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.627202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.22.157.252:44755)
0928 15:51:19.643281 (+ 16079us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:19.643287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:19.643308 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:51:19.643370 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:19.643375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:19.643460 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:19.644196 (+ 736us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:19.644492 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:19.647798 (+ 3306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:19.651371 (+ 3573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:19.652691 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:19.652880 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:19.652990 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:19.653304 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 15:51:19.653470 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":15919,"thread_start_us":16444,"threads_started":1}
I20260928 15:51:19.654328 30743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.637714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38326 (local address 127.22.157.253:45249)
0928 15:51:19.641738 (+ 4024us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:19.641749 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:19.641777 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:51:19.641873 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:19.641882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:19.641988 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:19.642144 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:19.643094 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:19.644437 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:19.651372 (+ 6935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:19.652671 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:19.652882 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:19.652963 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:19.653304 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 15:51:19.653464 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3828,"thread_start_us":4181,"threads_started":1}
I20260928 15:51:19.654574 30194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e588dbb8beb24471ac35e7ee90a05417" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "380d381cd4b44b80bafd3a5fc51beb24" is_pre_election: true
I20260928 15:51:19.654871 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "380d381cd4b44b80bafd3a5fc51beb24" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e588dbb8beb24471ac35e7ee90a05417" is_pre_election: true
I20260928 15:51:19.654995 30194 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e588dbb8beb24471ac35e7ee90a05417 in term 1.
I20260928 15:51:19.655220 30005 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 380d381cd4b44b80bafd3a5fc51beb24 in term 1.
I20260928 15:51:19.655496 29980 leader_election.cc:302] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 380d381cd4b44b80bafd3a5fc51beb24, e588dbb8beb24471ac35e7ee90a05417; no voters: 4d59f65974d44666b7524a34baf610b7
I20260928 15:51:19.655794 30736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 15:51:19.655926 30736 raft_consensus.cc:482] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d59f65974d44666b7524a34baf610b7)
I20260928 15:51:19.656059 30736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:51:19.656200 30170 leader_election.cc:302] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 380d381cd4b44b80bafd3a5fc51beb24, e588dbb8beb24471ac35e7ee90a05417; no voters: 4d59f65974d44666b7524a34baf610b7
I20260928 15:51:19.656708 30737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 15:51:19.656865 30737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d59f65974d44666b7524a34baf610b7)
I20260928 15:51:19.656965 30737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:51:19.659441 30736 raft_consensus.cc:504] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } }
I20260928 15:51:19.660123 30737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } }
I20260928 15:51:19.660326 30736 leader_election.cc:288] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 2 election: Requested vote from peers 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257), 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252:44755)
I20260928 15:51:19.660992 30737 leader_election.cc:288] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [CANDIDATE]: Term 2 election: Requested vote from peers 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257), e588dbb8beb24471ac35e7ee90a05417 (127.22.157.253:45249)
I20260928 15:51:19.661037 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.660494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51340)
0928 15:51:19.660708 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.660843 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 15:51:19.661475 30194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e588dbb8beb24471ac35e7ee90a05417" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "380d381cd4b44b80bafd3a5fc51beb24"
I20260928 15:51:19.661690 30005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "380d381cd4b44b80bafd3a5fc51beb24" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e588dbb8beb24471ac35e7ee90a05417"
I20260928 15:51:19.661840 30194 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e588dbb8beb24471ac35e7ee90a05417 in current term 2: Already voted for candidate 380d381cd4b44b80bafd3a5fc51beb24 in this term.
I20260928 15:51:19.662012 30005 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 380d381cd4b44b80bafd3a5fc51beb24 in current term 2: Already voted for candidate e588dbb8beb24471ac35e7ee90a05417 in this term.
I20260928 15:51:19.662349 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.661081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51350)
0928 15:51:19.662056 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.662156 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 15:51:19.662604 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.661993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51358)
0928 15:51:19.662263 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.662331 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
W20260928 15:51:19.662817 29981 leader_election.cc:334] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257): Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:19.662954 29981 leader_election.cc:302] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e588dbb8beb24471ac35e7ee90a05417; no voters: 380d381cd4b44b80bafd3a5fc51beb24, 4d59f65974d44666b7524a34baf610b7
I20260928 15:51:19.663250 30736 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 15:51:19.663535 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:19.663164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51370)
0928 15:51:19.663299 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:19.663419 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 15:51:19.663712 30171 leader_election.cc:334] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257): Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:19.663893 30171 leader_election.cc:302] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 380d381cd4b44b80bafd3a5fc51beb24; no voters: 4d59f65974d44666b7524a34baf610b7, e588dbb8beb24471ac35e7ee90a05417
I20260928 15:51:19.664173 30737 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 15:51:20.105562 30746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:20.104579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.193:51033)
0928 15:51:20.105172 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:20.105302 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":325,"threads_started":1}
I20260928 15:51:20.109093 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:20.108099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.195:59135)
0928 15:51:20.108680 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:20.108845 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":296,"threads_started":1}
I20260928 15:51:20.114410 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:20.113533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:39291)
0928 15:51:20.114144 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:20.114249 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":365,"threads_started":1}
I20260928 15:51:21.107517 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.106724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.193:51007)
0928 15:51:21.107253 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.107352 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":296,"threads_started":1}
I20260928 15:51:21.110852 30750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.109930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.195:38363)
0928 15:51:21.110478 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.110615 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":307,"threads_started":1}
I20260928 15:51:21.116066 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.115169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:38747)
0928 15:51:21.115693 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.115806 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":289,"threads_started":1}
I20260928 15:51:21.858686 30752 raft_consensus.cc:482] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:21.858999 30752 raft_consensus.cc:504] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } }
I20260928 15:51:21.859763 30752 leader_election.cc:288] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257), 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252:44755)
I20260928 15:51:21.860443 30194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e588dbb8beb24471ac35e7ee90a05417" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "380d381cd4b44b80bafd3a5fc51beb24" is_pre_election: true
I20260928 15:51:21.860698 30194 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e588dbb8beb24471ac35e7ee90a05417 in term 2.
I20260928 15:51:21.860868 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.859963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51642)
0928 15:51:21.860438 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.860578 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":259,"threads_started":1}
I20260928 15:51:21.861269 29980 leader_election.cc:302] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380d381cd4b44b80bafd3a5fc51beb24, e588dbb8beb24471ac35e7ee90a05417; no voters:
I20260928 15:51:21.861567 30752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 15:51:21.861709 30752 raft_consensus.cc:482] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:21.861824 30752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:51:21.862463 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.862054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51652)
0928 15:51:21.862233 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.862321 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:51:21.862735 29981 leader_election.cc:334] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257): Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:21.864188 30752 raft_consensus.cc:504] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } }
I20260928 15:51:21.864780 30752 leader_election.cc:288] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 3 election: Requested vote from peers 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257), 380d381cd4b44b80bafd3a5fc51beb24 (127.22.157.252:44755)
I20260928 15:51:21.865288 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.864906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51654)
0928 15:51:21.865045 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.865130 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 15:51:21.865288 30194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e588dbb8beb24471ac35e7ee90a05417" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "380d381cd4b44b80bafd3a5fc51beb24"
I20260928 15:51:21.865545 30194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:51:21.866211 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.865862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51664)
0928 15:51:21.865999 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.866076 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 15:51:21.866384 29981 leader_election.cc:334] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257): Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
I20260928 15:51:21.867893 30194 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e588dbb8beb24471ac35e7ee90a05417 in term 3.
I20260928 15:51:21.868301 29980 leader_election.cc:302] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 380d381cd4b44b80bafd3a5fc51beb24, e588dbb8beb24471ac35e7ee90a05417; no voters: 4d59f65974d44666b7524a34baf610b7
I20260928 15:51:21.868613 30752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:51:21.869020 30752 raft_consensus.cc:686] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [term 3 LEADER]: Becoming Leader. State: Replica: e588dbb8beb24471ac35e7ee90a05417, State: Running, Role: LEADER
I20260928 15:51:21.869406 30752 consensus_queue.cc:237] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } }
I20260928 15:51:21.871918 30755 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e588dbb8beb24471ac35e7ee90a05417. Latest consensus state: current_term: 3 leader_uuid: "e588dbb8beb24471ac35e7ee90a05417" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:21.872217 30755 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:21.872884 30757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:21.876391 30757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:21.877293 30757 catalog_manager.cc:1259] Loaded cluster ID: 5b1dfd5385a4479081dbef524155c4f6
I20260928 15:51:21.877362 30757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:21.878474 30757 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:21.879251 30757 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Loaded TSK: 0
I20260928 15:51:21.880055 30757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:51:21.899461 29994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:51:21.906581 29994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:51:21.910910 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.910426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51686)
0928 15:51:21.910611 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.910715 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 15:51:21.911113 30194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [term 3 FOLLOWER]: Refusing update from remote peer e588dbb8beb24471ac35e7ee90a05417: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 15:51:21.911963 30755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 15:51:21.912222 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.911655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51692)
0928 15:51:21.911849 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.911953 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 15:51:21.912459 29981 consensus_peers.cc:487] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 -> Peer 4d59f65974d44666b7524a34baf610b7 (127.22.157.254:46257): Couldn't send request to peer 4d59f65974d44666b7524a34baf610b7. Status: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 15:51:21.914480 30759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e588dbb8beb24471ac35e7ee90a05417. Latest consensus state: current_term: 3 leader_uuid: "e588dbb8beb24471ac35e7ee90a05417" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:21.914793 30759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:51:21.917212 30759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e588dbb8beb24471ac35e7ee90a05417" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:21.917496 30759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380d381cd4b44b80bafd3a5fc51beb24 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:51:21.918602 30765 sys_catalog.cc:455] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e588dbb8beb24471ac35e7ee90a05417" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4d59f65974d44666b7524a34baf610b7" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46257 } } peers { permanent_uuid: "e588dbb8beb24471ac35e7ee90a05417" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 45249 } attrs { promote: false } } peers { permanent_uuid: "380d381cd4b44b80bafd3a5fc51beb24" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44755 } attrs { promote: false } } }
I20260928 15:51:21.919073 30765 sys_catalog.cc:458] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:21.938984 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.929178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:38803 (local address 127.0.0.1:39888)
0928 15:51:21.929826 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.929846 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:21.929952 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:21.930708 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:21.930713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:21.930742 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:21.931203 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:21.931219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.933334 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.933343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:21.936219 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:21.936243 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.938112 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.938119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:21.938322 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:21.938358 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:51:21.938472 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 15:51:21.938616 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":1410,"threads_started":1}
I20260928 15:51:21.939819 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.929919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.22.157.193:38803)
0928 15:51:21.930384 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:21.930389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:21.930407 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:51:21.930478 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:21.930483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:21.930562 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:21.931431 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:21.931645 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.933145 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.936498 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.937730 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.937897 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:21.937977 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:21.939352 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0928 15:51:21.939528 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":268,"threads_started":1}
I20260928 15:51:21.940527 30301 tablet_service.cc:1467] Processing CreateTablet for tablet 41102040d81f4dc2ad3599a45f31cd31 (DEFAULT_TABLE table=test-table [id=069ef737a6ac417598833dfb685fd522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:21.941001 30301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41102040d81f4dc2ad3599a45f31cd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:21.944185 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.928847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:32891 (local address 127.0.0.1:53398)
0928 15:51:21.929006 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.929028 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:21.929176 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:21.937549 (+ 8373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:21.937554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:21.937590 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:21.937949 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:21.937962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.939344 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.939349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:21.941232 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:21.941246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.943108 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.943118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:21.943398 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:21.943454 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 15:51:21.943580 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 15:51:21.943771 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 15:51:21.945045 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.929531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.22.157.194:32891)
0928 15:51:21.937173 (+ 7642us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:21.937182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:21.937207 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:51:21.937289 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:21.937297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:21.937398 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:21.937535 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:21.938144 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.939182 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.941649 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.942865 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.943014 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:21.943086 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:21.944438 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0928 15:51:21.944666 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7477,"thread_start_us":7731,"threads_started":1}
I20260928 15:51:21.945716 30448 tablet_service.cc:1467] Processing CreateTablet for tablet 41102040d81f4dc2ad3599a45f31cd31 (DEFAULT_TABLE table=test-table [id=069ef737a6ac417598833dfb685fd522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:21.946187 30448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41102040d81f4dc2ad3599a45f31cd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:21.947999 30771 tablet_bootstrap.cc:492] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: Bootstrap starting.
I20260928 15:51:21.949982 30771 tablet_bootstrap.cc:654] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:21.952339 30771 tablet_bootstrap.cc:492] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: No bootstrap required, opened a new log
I20260928 15:51:21.952499 30771 ts_tablet_manager.cc:1397] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 15:51:21.952778 30773 tablet_bootstrap.cc:492] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: Bootstrap starting.
I20260928 15:51:21.953387 30771 raft_consensus.cc:348] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } }
I20260928 15:51:21.953637 30771 raft_consensus.cc:374] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:21.953728 30771 raft_consensus.cc:729] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09dac1b78ef4becac2379eaf9bef297, State: Initialized, Role: FOLLOWER
I20260928 15:51:21.954154 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.930070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.22.157.195:45159)
0928 15:51:21.945109 (+ 15039us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:21.945117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:21.945137 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:51:21.945599 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:21.945604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:21.945686 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:21.945792 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:21.946448 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.948935 (+ 2487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.950858 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.952068 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.953336 (+ 1268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:21.953433 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:21.953637 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 15:51:21.953804 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14910,"thread_start_us":14805,"threads_started":1}
I20260928 15:51:21.954160 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.929693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:45159 (local address 127.0.0.1:37490)
0928 15:51:21.938207 (+ 8514us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.938231 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:21.938373 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:21.945837 (+ 7464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:21.945842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:21.945932 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:21.946274 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:21.946286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.949081 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.949087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:21.950708 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:21.950720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:21.953245 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:21.953252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:21.953440 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:21.953481 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 15:51:21.953607 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 15:51:21.953782 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8353,"thread_start_us":8188,"threads_started":1}
I20260928 15:51:21.953987 30771 consensus_queue.cc:260] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } }
I20260928 15:51:21.954664 30773 tablet_bootstrap.cc:654] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:21.955163 30593 tablet_service.cc:1467] Processing CreateTablet for tablet 41102040d81f4dc2ad3599a45f31cd31 (DEFAULT_TABLE table=test-table [id=069ef737a6ac417598833dfb685fd522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:21.955628 30593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41102040d81f4dc2ad3599a45f31cd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:21.957149 30773 tablet_bootstrap.cc:492] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: No bootstrap required, opened a new log
I20260928 15:51:21.957293 30773 ts_tablet_manager.cc:1397] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 15:51:21.957762 30771 ts_tablet_manager.cc:1428] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 15:51:21.959123 30773 raft_consensus.cc:348] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } }
I20260928 15:51:21.959362 30773 raft_consensus.cc:374] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:21.959458 30773 raft_consensus.cc:729] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bda64e44c8f4422ad664c137df59822, State: Initialized, Role: FOLLOWER
I20260928 15:51:21.959725 30773 consensus_queue.cc:260] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } }
I20260928 15:51:21.962248 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.957907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.193:45813)
0928 15:51:21.961810 (+ 3903us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.961958 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3756,"thread_start_us":3618,"threads_started":1}
I20260928 15:51:21.963909 30778 tablet_bootstrap.cc:492] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: Bootstrap starting.
I20260928 15:51:21.965637 30778 tablet_bootstrap.cc:654] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:21.967989 30778 tablet_bootstrap.cc:492] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: No bootstrap required, opened a new log
I20260928 15:51:21.968153 30778 ts_tablet_manager.cc:1397] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 15:51:21.968873 30778 raft_consensus.cc:348] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } }
I20260928 15:51:21.969080 30778 raft_consensus.cc:374] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:21.969416 30778 raft_consensus.cc:729] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b3c169d6b8401eb1d32fa23e69d8c2, State: Initialized, Role: FOLLOWER
I20260928 15:51:21.969730 30778 consensus_queue.cc:260] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } }
I20260928 15:51:21.970232 30773 ts_tablet_manager.cc:1428] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260928 15:51:21.974745 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.964090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.193:54561)
0928 15:51:21.974362 (+ 10272us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.974503 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10116,"threads_started":1}
I20260928 15:51:21.974988 30517 heartbeater.cc:503] Master 127.22.157.253:45249 was elected leader, sending a full tablet report...
I20260928 15:51:21.981580 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.966867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:34449)
0928 15:51:21.981258 (+ 14391us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.981372 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14262,"thread_start_us":14111,"threads_started":1}
I20260928 15:51:21.982292 30778 ts_tablet_manager.cc:1428] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260928 15:51:21.983536 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.982941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:48553)
0928 15:51:21.983238 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.983345 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260928 15:51:21.984452 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.982529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.195:47647)
0928 15:51:21.984105 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.984216 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1468,"thread_start_us":1316,"threads_started":1}
I20260928 15:51:21.984529 30660 heartbeater.cc:503] Master 127.22.157.253:45249 was elected leader, sending a full tablet report...
I20260928 15:51:21.985666 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:21.985130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.195:57217)
0928 15:51:21.985370 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:21.985454 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260928 15:51:22.002213 30777 raft_consensus.cc:482] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:22.002481 30777 raft_consensus.cc:504] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } }
I20260928 15:51:22.003314 30777 leader_election.cc:288] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803), d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159)
I20260928 15:51:22.003827 30613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41102040d81f4dc2ad3599a45f31cd31" candidate_uuid: "0bda64e44c8f4422ad664c137df59822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" is_pre_election: true
I20260928 15:51:22.003836 30321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41102040d81f4dc2ad3599a45f31cd31" candidate_uuid: "0bda64e44c8f4422ad664c137df59822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09dac1b78ef4becac2379eaf9bef297" is_pre_election: true
I20260928 15:51:22.004065 30613 raft_consensus.cc:2454] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bda64e44c8f4422ad664c137df59822 in term 0.
I20260928 15:51:22.004091 30321 raft_consensus.cc:2454] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bda64e44c8f4422ad664c137df59822 in term 0.
I20260928 15:51:22.004554 30400 leader_election.cc:302] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bda64e44c8f4422ad664c137df59822, d09dac1b78ef4becac2379eaf9bef297; no voters:
I20260928 15:51:22.004887 30777 raft_consensus.cc:2781] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:51:22.005013 30777 raft_consensus.cc:482] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:22.005087 30777 raft_consensus.cc:3037] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:22.007273 30777 raft_consensus.cc:504] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } }
I20260928 15:51:22.007854 30777 leader_election.cc:288] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [CANDIDATE]: Term 1 election: Requested vote from peers d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803), d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159)
I20260928 15:51:22.008435 30613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41102040d81f4dc2ad3599a45f31cd31" candidate_uuid: "0bda64e44c8f4422ad664c137df59822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2"
I20260928 15:51:22.008450 30321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41102040d81f4dc2ad3599a45f31cd31" candidate_uuid: "0bda64e44c8f4422ad664c137df59822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09dac1b78ef4becac2379eaf9bef297"
I20260928 15:51:22.008661 30321 raft_consensus.cc:3037] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:22.008694 30613 raft_consensus.cc:3037] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:22.010896 30613 raft_consensus.cc:2454] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bda64e44c8f4422ad664c137df59822 in term 1.
I20260928 15:51:22.011492 30401 leader_election.cc:302] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bda64e44c8f4422ad664c137df59822, d7b3c169d6b8401eb1d32fa23e69d8c2; no voters:
I20260928 15:51:22.011880 30321 raft_consensus.cc:2454] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bda64e44c8f4422ad664c137df59822 in term 1.
I20260928 15:51:22.012512 30777 raft_consensus.cc:2781] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:22.013105 30777 raft_consensus.cc:686] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 1 LEADER]: Becoming Leader. State: Replica: 0bda64e44c8f4422ad664c137df59822, State: Running, Role: LEADER
I20260928 15:51:22.013614 30777 consensus_queue.cc:237] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } }
I20260928 15:51:22.016918 29995 catalog_manager.cc:5495] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bda64e44c8f4422ad664c137df59822 (127.22.157.194). New cstate: current_term: 1 leader_uuid: "0bda64e44c8f4422ad664c137df59822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bda64e44c8f4422ad664c137df59822" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 32891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 } health_report { overall_health: UNKNOWN } } }
I20260928 15:51:22.025169 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.013861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:33607)
0928 15:51:22.024887 (+ 11026us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:22.024996 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10900}
I20260928 15:51:22.050359 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.045419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.22.157.194:32891)
0928 15:51:22.045644 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:22.045649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:22.045667 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:51:22.045729 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:22.045732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:22.045796 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:22.045871 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:51:22.046235 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:22.047029 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:22.048464 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:22.049000 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:22.049107 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:22.049670 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:51:22.049790 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:51:22.049863 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:22.049989 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 15:51:22.050060 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:51:22.053774 30613 raft_consensus.cc:1261] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 1 FOLLOWER]: Refusing update from remote peer 0bda64e44c8f4422ad664c137df59822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:22.053820 30321 raft_consensus.cc:1261] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [term 1 FOLLOWER]: Refusing update from remote peer 0bda64e44c8f4422ad664c137df59822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:22.054493 30784 consensus_queue.cc:1035] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b3c169d6b8401eb1d32fa23e69d8c2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:22.054824 30777 consensus_queue.cc:1035] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09dac1b78ef4becac2379eaf9bef297" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:22.056331 30369 heartbeater.cc:503] Master 127.22.157.253:45249 was elected leader, sending a full tablet report...
I20260928 15:51:22.059142 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.057792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.193:40785)
0928 15:51:22.058582 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:22.058809 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560}
I20260928 15:51:22.059839 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.058205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.195:45821)
0928 15:51:22.059459 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:22.059616 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1100}
I20260928 15:51:22.066392 30779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.065815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.193:43075)
0928 15:51:22.066123 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:22.066222 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260928 15:51:22.067622 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.066036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:53455)
0928 15:51:22.067343 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:22.067444 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1201}
I20260928 15:51:22.067694 30788 mvcc.cc:205] Tried to move back new op lower bound from 7334341353686216704 to 7334341353530503168. Current Snapshot: MvccSnapshot[applied={T|T < 7334341353686216704}]
I20260928 15:51:22.069253 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.068184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.194:47691)
0928 15:51:22.068997 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:22.069099 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":713}
I20260928 15:51:22.069252 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.068801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.22.157.195:49187)
0928 15:51:22.069034 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:22.069113 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 15:51:22.075892 29995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:22.076149 29995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:22.081597 29995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e588dbb8beb24471ac35e7ee90a05417: Sending DeleteTablet for 3 replicas of tablet 41102040d81f4dc2ad3599a45f31cd31
I20260928 15:51:22.082548 30448 tablet_service.cc:1514] Processing DeleteTablet for tablet 41102040d81f4dc2ad3599a45f31cd31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:22 UTC) from {username='oryx', principal='oryx/127.22.157.253@KRBTEST.COM'} at 127.0.0.1:53398
I20260928 15:51:22.082827 30301 tablet_service.cc:1514] Processing DeleteTablet for tablet 41102040d81f4dc2ad3599a45f31cd31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:22 UTC) from {username='oryx', principal='oryx/127.22.157.253@KRBTEST.COM'} at 127.0.0.1:39888
I20260928 15:51:22.082828 30593 tablet_service.cc:1514] Processing DeleteTablet for tablet 41102040d81f4dc2ad3599a45f31cd31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:22 UTC) from {username='oryx', principal='oryx/127.22.157.253@KRBTEST.COM'} at 127.0.0.1:37490
I20260928 15:51:22.083235 30795 tablet_replica.cc:330] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: stopping tablet replica
I20260928 15:51:22.083442 30795 raft_consensus.cc:2229] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:51:22.083890 30795 raft_consensus.cc:2258] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:22.085090 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 30161
I20260928 15:51:22.085503 30795 ts_tablet_manager.cc:1905] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:22.095708 30795 ts_tablet_manager.cc:1918] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:22.095942 30795 log.cc:1196] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/41102040d81f4dc2ad3599a45f31cd31
I20260928 15:51:22.096419 30797 tablet_replica.cc:330] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: stopping tablet replica
I20260928 15:51:22.096500 30795 ts_tablet_manager.cc:1939] T 41102040d81f4dc2ad3599a45f31cd31 P 0bda64e44c8f4422ad664c137df59822: Deleting consensus metadata
I20260928 15:51:22.096654 30797 raft_consensus.cc:2229] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:22.097086 30797 raft_consensus.cc:2258] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:22.098093 29978 catalog_manager.cc:4841] TS 0bda64e44c8f4422ad664c137df59822 (127.22.157.194:32891): tablet 41102040d81f4dc2ad3599a45f31cd31 (table test-table [id=069ef737a6ac417598833dfb685fd522]) successfully deleted
I20260928 15:51:22.098549 30797 ts_tablet_manager.cc:1905] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:22.100366 30796 tablet_replica.cc:330] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: stopping tablet replica
I20260928 15:51:22.100589 30796 raft_consensus.cc:2229] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:22.100986 30796 raft_consensus.cc:2258] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:22.102303 30796 ts_tablet_manager.cc:1905] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:22.107220 30796 ts_tablet_manager.cc:1918] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:22.107367 30796 log.cc:1196] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/41102040d81f4dc2ad3599a45f31cd31
I20260928 15:51:22.107784 30796 ts_tablet_manager.cc:1939] T 41102040d81f4dc2ad3599a45f31cd31 P d09dac1b78ef4becac2379eaf9bef297: Deleting consensus metadata
I20260928 15:51:22.108858 29978 catalog_manager.cc:4841] TS d09dac1b78ef4becac2379eaf9bef297 (127.22.157.193:38803): tablet 41102040d81f4dc2ad3599a45f31cd31 (table test-table [id=069ef737a6ac417598833dfb685fd522]) successfully deleted
I20260928 15:51:22.111884 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.086833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.22.157.193:38803)
0928 15:51:22.087097 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:22.087107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:22.096685 (+ 9578us) server_negotiation.cc:407] Connection header received
0928 15:51:22.097040 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:22.097046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:22.097137 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:22.097224 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:22.099151 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:22.099908 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:22.101479 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:22.102131 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:22.102274 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:22.105140 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:22.105160 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:22.105191 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:22.105252 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:22.108767 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:22.109627 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:22.109638 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:22.109641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:22.109795 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:22.110184 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:22.110190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:22.110193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:22.110593 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:22.110970 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:22.111264 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 15:51:22.111518 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:51:22.112051 30797 ts_tablet_manager.cc:1918] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:22.112200 30797 log.cc:1196] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/41102040d81f4dc2ad3599a45f31cd31
I20260928 15:51:22.112669 30797 ts_tablet_manager.cc:1939] T 41102040d81f4dc2ad3599a45f31cd31 P d7b3c169d6b8401eb1d32fa23e69d8c2: Deleting consensus metadata
I20260928 15:51:22.112704 30362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:22.114102 29979 catalog_manager.cc:4841] TS d7b3c169d6b8401eb1d32fa23e69d8c2 (127.22.157.195:45159): tablet 41102040d81f4dc2ad3599a45f31cd31 (table test-table [id=069ef737a6ac417598833dfb685fd522]) successfully deleted
I20260928 15:51:22.208667 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 30161
I20260928 15:51:22.227308 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 30379
I20260928 15:51:22.241842 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.228099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.22.157.194:32891)
0928 15:51:22.228297 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:22.228301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:22.228321 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:51:22.228456 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:22.228460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:22.228573 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:22.228659 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:22.230045 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:22.230828 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:22.232109 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:22.232532 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:22.232653 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:22.235402 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:22.235420 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:22.235436 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:22.235477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:22.238653 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:22.239457 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:22.239465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:22.239467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:22.239645 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:22.240008 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:22.240011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:22.240013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:22.240381 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:22.240637 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:22.241245 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 15:51:22.241429 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:51:22.242588 30510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:22.368162 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 30379
I20260928 15:51:22.391809 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 30530
I20260928 15:51:22.405570 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.392689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.22.157.195:45159)
0928 15:51:22.392927 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:22.392931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:22.392965 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:51:22.393116 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:22.393120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:22.393197 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:22.393284 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:22.394665 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:22.395399 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:22.396605 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:22.397108 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:22.397218 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:22.399805 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:22.399821 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:22.399837 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:22.399872 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:22.402853 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:22.403531 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:22.403539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:22.403542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:22.403685 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:22.404004 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:22.404009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:22.404011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:22.404371 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:22.404625 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:22.404963 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 15:51:22.405154 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:51:22.406296 30653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:22.503355 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 30530
I20260928 15:51:22.506713 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.506009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51696)
0928 15:51:22.506448 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:22.506539 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
I20260928 15:51:22.508035 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:22.507695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46257 (local address 127.0.0.1:51700)
0928 15:51:22.507854 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:22.507921 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:46257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 15:51:22.522286 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 29854
I20260928 15:51:22.523399 30025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:22.648458 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 29854
W20260928 15:51:22.670181 30168 connection.cc:535] server connection from 127.0.0.1:47618 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:51:22.670616 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 29957
I20260928 15:51:22.671887 30215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:22.791671 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 29957
2026-09-28T15:51:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12236 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:51:22 dist-test-slave-1vw0 krb5kdc[30808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:51:22 dist-test-slave-1vw0 krb5kdc[30808](info): set up 2 sockets
Sep 28 15:51:22 dist-test-slave-1vw0 krb5kdc[30808](info): commencing operation
krb5kdc: starting...
W20260928 15:51:23.958062 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:51:24 dist-test-slave-1vw0 krb5kdc[30808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:51:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.254.keytab.
Entry for principal oryx/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.254.keytab.
I20260928 15:51:24.219412 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:41669
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.254.keytab
--principal=oryx/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:41669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:24.566025 30821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:24.566329 30821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:24.566427 30821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:24.573889 30821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:24.574009 30821 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 15:51:24.574050 30821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:24.574105 30821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:24.574172 30821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:24.574229 30821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:24.584805 30821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:41669
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.254.keytab
--principal=oryx/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:41669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:24.586145 30821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:24.587564 30821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:24.600495 30826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:24.600548 30827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:24.600544 30829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:24.601765 30821 server_base.cc:1034] running on GCE node
I20260928 15:51:24.602661 30821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:24.604279 30821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:24.605548 30821 hybrid_clock.cc:648] HybridClock initialized: now 1790610684605475 us; error 36 us; skew 500 ppm
I20260928 15:51:24.605974 30821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:24 dist-test-slave-1vw0 krb5kdc[30808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610684, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:24.609741 30821 init.cc:377] Logged in from keytab as oryx/127.22.157.254@KRBTEST.COM (short username oryx)
I20260928 15:51:24.611632 30821 webserver.cc:460] Webserver started at http://127.22.157.254:46071/ using document root <none> and password file <none>
I20260928 15:51:24.612454 30821 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:24.612612 30821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:24.613023 30821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:24.616320 30821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f"
format_stamp: "Formatted at 2026-09-28 15:51:24 on dist-test-slave-1vw0"
server_key: "aab70c17864f1ed479f3c17a09f30917"
server_key_iv: "a4e23e1575c0f2c1b3e0093714a6cfa8"
server_key_version: "encryptionkey@0"
I20260928 15:51:24.617277 30821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f"
format_stamp: "Formatted at 2026-09-28 15:51:24 on dist-test-slave-1vw0"
server_key: "aab70c17864f1ed479f3c17a09f30917"
server_key_iv: "a4e23e1575c0f2c1b3e0093714a6cfa8"
server_key_version: "encryptionkey@0"
I20260928 15:51:24.623806 30821 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:51:24.628319 30836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:24.630132 30821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:51:24.630360 30821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f"
format_stamp: "Formatted at 2026-09-28 15:51:24 on dist-test-slave-1vw0"
server_key: "aab70c17864f1ed479f3c17a09f30917"
server_key_iv: "a4e23e1575c0f2c1b3e0093714a6cfa8"
server_key_version: "encryptionkey@0"
I20260928 15:51:24.630627 30821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:24.648047 30821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:24.652987 30821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:24.653335 30821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:24.676898 30821 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:41669
I20260928 15:51:24.676908 30898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:41669 every 8 connection(s)
I20260928 15:51:24.678381 30821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:24.682313 30899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:24.684273 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 30821
I20260928 15:51:24.684489 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:24.685357 23159 external_mini_cluster.cc:1428] Setting key 809d263dac6534fe53d9eb5023d9233d
I20260928 15:51:24.696162 30899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f: Bootstrap starting.
I20260928 15:51:24.700475 30899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f: Neither blocks nor log segments found. Creating new log.
Sep 28 15:51:24 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:24.701715 30899 log.cc:824] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:24.704895 30899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f: No bootstrap required, opened a new log
I20260928 15:51:24.710300 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:24.687578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.22.157.254:41669)
0928 15:51:24.688471 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:24.688484 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:24.688541 (+ 57us) server_negotiation.cc:407] Connection header received
0928 15:51:24.690428 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:24.690453 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:24.690904 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:24.691304 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:24.693212 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:24.694687 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:24.696553 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:24.697351 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:24.697511 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:24.702407 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:24.702417 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:24.702433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:24.702481 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:24.705826 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:24.706620 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:24.706628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:24.706635 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:24.706814 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:24.707243 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:24.707247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:24.707252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:24.707905 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:24.708241 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:24.708671 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 15:51:24.709202 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":429,"threads_started":1}
I20260928 15:51:24.713573 30899 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41669 } }
I20260928 15:51:24.714015 30899 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:24.714136 30899 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8001fd5dbb4c7d85b99072bf3ecf9f, State: Initialized, Role: FOLLOWER
I20260928 15:51:24.714751 30899 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41669 } }
I20260928 15:51:24.714993 30899 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:24.715154 30899 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:24.715328 30899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:24.717696 30899 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41669 } }
I20260928 15:51:24.718281 30899 leader_election.cc:302] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd8001fd5dbb4c7d85b99072bf3ecf9f; no voters:
I20260928 15:51:24.719110 30899 leader_election.cc:288] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:24.719256 30904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:24.719951 30904 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [term 1 LEADER]: Becoming Leader. State: Replica: bd8001fd5dbb4c7d85b99072bf3ecf9f, State: Running, Role: LEADER
I20260928 15:51:24.720526 30904 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41669 } }
I20260928 15:51:24.721556 30899 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:24.724431 30905 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41669 } } }
I20260928 15:51:24.724900 30905 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:24.724402 30906 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd8001fd5dbb4c7d85b99072bf3ecf9f. Latest consensus state: current_term: 1 leader_uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd8001fd5dbb4c7d85b99072bf3ecf9f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41669 } } }
I20260928 15:51:24.725730 30906 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:24.727236 30912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:24.732966 30912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:24.742647 30912 catalog_manager.cc:1347] Generated new cluster ID: ccc76e81c72f4a58beb14fd1580bc5d6
I20260928 15:51:24.742805 30912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:24.750085 30912 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:24.751423 30912 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:24.760200 30912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f: Generated new TSK 0
I20260928 15:51:24.761140 30912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab.
Entry for principal oryx/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab.
I20260928 15:51:24.829299 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:41669
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab
--principal=oryx/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:25.177004 30925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:25.177299 30925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:25.177397 30925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:25.184661 30925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:25.184806 30925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:25.184893 30925 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:51:25.184975 30925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:25.194975 30925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab
--principal=oryx/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.157.254:41669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:25.196507 30925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:25.197956 30925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:25.213297 30930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:25.213335 30931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:25.213377 30933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:25.213920 30925 server_base.cc:1034] running on GCE node
I20260928 15:51:25.214850 30925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:25.216068 30925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:25.217401 30925 hybrid_clock.cc:648] HybridClock initialized: now 1790610685217236 us; error 97 us; skew 500 ppm
I20260928 15:51:25.217806 30925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:25 dist-test-slave-1vw0 krb5kdc[30808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610685, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:25.221946 30925 init.cc:377] Logged in from keytab as oryx/127.22.157.193@KRBTEST.COM (short username oryx)
I20260928 15:51:25.223718 30925 webserver.cc:460] Webserver started at http://127.22.157.193:45879/ using document root <none> and password file <none>
I20260928 15:51:25.224689 30925 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:25.225030 30925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:25.225409 30925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:25.229250 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "1c7fa319f5ab40069abf57f70b3bb7cb"
format_stamp: "Formatted at 2026-09-28 15:51:25 on dist-test-slave-1vw0"
server_key: "08aebbfb486ab86565c5a41fa0764d8b"
server_key_iv: "8879928882cffab0825fcf61830c2126"
server_key_version: "encryptionkey@0"
I20260928 15:51:25.230149 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "1c7fa319f5ab40069abf57f70b3bb7cb"
format_stamp: "Formatted at 2026-09-28 15:51:25 on dist-test-slave-1vw0"
server_key: "08aebbfb486ab86565c5a41fa0764d8b"
server_key_iv: "8879928882cffab0825fcf61830c2126"
server_key_version: "encryptionkey@0"
I20260928 15:51:25.237011 30925 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 15:51:25.241113 30940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:25.242683 30925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:25.242892 30925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "1c7fa319f5ab40069abf57f70b3bb7cb"
format_stamp: "Formatted at 2026-09-28 15:51:25 on dist-test-slave-1vw0"
server_key: "08aebbfb486ab86565c5a41fa0764d8b"
server_key_iv: "8879928882cffab0825fcf61830c2126"
server_key_version: "encryptionkey@0"
I20260928 15:51:25.243166 30925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:25.264135 30925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:25.269163 30925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:25.269464 30925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:25.272328 30925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:25.272462 30925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:25.272619 30925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:25.272696 30925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:51:25 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610685, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:25.298811 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:25.273280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:47419 (local address 127.22.157.254:41669)
0928 15:51:25.273806 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:25.273814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:25.275172 (+ 1358us) server_negotiation.cc:407] Connection header received
0928 15:51:25.277657 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:25.277663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:25.277778 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:25.277906 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:25.280632 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.281740 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.284526 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.285216 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.285394 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:25.291201 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:25.291207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:25.291210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:25.291250 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:25.294842 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:25.296145 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:25.296150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:25.296152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:25.296285 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:25.296827 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:25.296832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:25.296836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:25.297143 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:25.297324 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:25.298149 (+ 825us) server_negotiation.cc:299] Negotiation successful
0928 15:51:25.298406 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":258,"threads_started":1}
I20260928 15:51:25.300467 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:25.274005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:41669 (local address 127.22.157.193:47419)
0928 15:51:25.274953 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:25.275013 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:25.277294 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:25.278324 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:25.278341 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:25.279323 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:25.280382 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:25.280415 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.281969 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.281973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:25.284355 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:25.284369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.285765 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.285771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:25.285980 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:25.286782 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:25.286827 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:25.290858 (+ 4031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:25.295047 (+ 4189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:25.295065 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:25.295099 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:25.295932 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:25.296417 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:25.296427 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:25.296429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:25.296663 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:25.297292 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:25.297302 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:25.297737 (+ 435us) client_negotiation.cc:769] Sending connection context
0928 15:51:25.298023 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 15:51:25.298665 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":403,"threads_started":1}
I20260928 15:51:25.326215 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:25.308403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:42821 (local address 127.22.157.254:41669)
0928 15:51:25.308618 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:25.308626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:25.309748 (+ 1122us) server_negotiation.cc:407] Connection header received
0928 15:51:25.310154 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:25.310163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:25.310275 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:25.310402 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:25.312004 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.312810 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.315013 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.315718 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.315877 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:25.318745 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:25.318753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:25.318760 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:25.318810 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:25.322696 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:25.323790 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:25.323797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:25.323800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:25.323980 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:25.324547 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:25.324556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:25.324559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:25.324930 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:25.325107 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:25.325593 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 15:51:25.325855 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:51:25.326479 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:25.308535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:41669 (local address 127.22.157.193:42821)
0928 15:51:25.309608 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:25.309632 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:25.309918 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:25.310589 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:25.310592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:25.311320 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:25.311829 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:25.311841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.312968 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.312981 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:25.314765 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:25.314787 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.316169 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.316174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:25.316371 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:25.317179 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:25.317202 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:25.318409 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:25.322877 (+ 4468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:25.322886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:25.322896 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:25.323574 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:25.324119 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:25.324124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:25.324128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:25.324363 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:25.325065 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:25.325073 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:25.325302 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 15:51:25.325464 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 15:51:25.325878 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":574,"threads_started":1}
I20260928 15:51:25.327047 30925 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:40665
I20260928 15:51:25.327068 31061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:40665 every 8 connection(s)
I20260928 15:51:25.328670 30925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:25.331115 31062 heartbeater.cc:347] Connected to a master server at 127.22.157.254:41669
I20260928 15:51:25.331349 31062 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:25.332001 31062 heartbeater.cc:511] Master 127.22.157.254:41669 requested a full tablet report, sending...
I20260928 15:51:25.334540 30854 ts_manager.cc:194] Registered new tserver with Master: 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665)
I20260928 15:51:25.336695 30854 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:47419
I20260928 15:51:25.338562 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 30925
I20260928 15:51:25.338769 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:51:25.339610 23159 external_mini_cluster.cc:1428] Setting key 228491d16240924f4fef8e358a5c67a1
WARNING: no policy specified for oryx/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab.
Entry for principal oryx/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab.
I20260928 15:51:25.392534 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:41669
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab
--principal=oryx/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:25.745062 31066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:25.745384 31066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:25.745551 31066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:25.753221 31066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:25.753346 31066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:25.753486 31066 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:51:25.753579 31066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:51:25.763800 31066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab
--principal=oryx/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.157.254:41669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:25.765251 31066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:25.766650 31066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:25.781487 31074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:25.781519 31072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:25.783155 31066 server_base.cc:1034] running on GCE node
W20260928 15:51:25.785038 31071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:25.786083 31066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:25.787351 31066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:25.788611 31066 hybrid_clock.cc:648] HybridClock initialized: now 1790610685788482 us; error 100 us; skew 500 ppm
I20260928 15:51:25.789039 31066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:25 dist-test-slave-1vw0 krb5kdc[30808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610685, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:25.792707 31066 init.cc:377] Logged in from keytab as oryx/127.22.157.194@KRBTEST.COM (short username oryx)
I20260928 15:51:25.794392 31066 webserver.cc:460] Webserver started at http://127.22.157.194:35703/ using document root <none> and password file <none>
I20260928 15:51:25.795168 31066 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:25.795285 31066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:25.795619 31066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:25.798820 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "bfb8e4be66de4b3d9e22145f08d9824f"
format_stamp: "Formatted at 2026-09-28 15:51:25 on dist-test-slave-1vw0"
server_key: "fe3be44351f36abfd5187a819c70f722"
server_key_iv: "0ae6fd1e9e14c14abe47298641a7796e"
server_key_version: "encryptionkey@0"
I20260928 15:51:25.799674 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "bfb8e4be66de4b3d9e22145f08d9824f"
format_stamp: "Formatted at 2026-09-28 15:51:25 on dist-test-slave-1vw0"
server_key: "fe3be44351f36abfd5187a819c70f722"
server_key_iv: "0ae6fd1e9e14c14abe47298641a7796e"
server_key_version: "encryptionkey@0"
I20260928 15:51:25.805608 31066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 15:51:25.809350 31081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:25.810739 31066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260928 15:51:25.810931 31066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "bfb8e4be66de4b3d9e22145f08d9824f"
format_stamp: "Formatted at 2026-09-28 15:51:25 on dist-test-slave-1vw0"
server_key: "fe3be44351f36abfd5187a819c70f722"
server_key_iv: "0ae6fd1e9e14c14abe47298641a7796e"
server_key_version: "encryptionkey@0"
I20260928 15:51:25.811210 31066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:25.832194 31066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:25.836809 31066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:25.837088 31066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:25.839876 31066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:25.839993 31066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:25.840217 31066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:25.840305 31066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:51:25 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610685, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:25.865885 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:25.840665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:39465 (local address 127.22.157.254:41669)
0928 15:51:25.841295 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:25.841301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:25.842496 (+ 1195us) server_negotiation.cc:407] Connection header received
0928 15:51:25.844632 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:25.844637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:25.844808 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:25.844981 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:25.848008 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.849175 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.851726 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.852547 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.852719 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:25.857851 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:25.857856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:25.857859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:25.857895 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:25.860973 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:25.862026 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:25.862034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:25.862038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:25.862221 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:25.862977 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:25.862982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:25.862985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:25.863289 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:25.863412 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:25.865061 (+ 1649us) server_negotiation.cc:299] Negotiation successful
0928 15:51:25.865366 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":343,"threads_started":1}
I20260928 15:51:25.868196 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:25.841391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:41669 (local address 127.22.157.194:39465)
0928 15:51:25.842306 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:25.842359 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:25.844328 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:25.845431 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:25.845448 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:25.846621 (+ 1173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:25.847809 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:25.847832 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.849372 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.849379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:25.851528 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:25.851549 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.852883 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.852893 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:25.853247 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:25.854126 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:25.854154 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:25.857588 (+ 3434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:25.861146 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:25.861159 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:25.861180 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:25.861877 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:25.862381 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:25.862390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:25.862397 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:25.862815 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:25.863444 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:25.863457 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:25.864176 (+ 719us) client_negotiation.cc:769] Sending connection context
0928 15:51:25.864647 (+ 471us) client_negotiation.cc:240] Negotiation successful
0928 15:51:25.865409 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":389,"threads_started":1}
I20260928 15:51:25.888944 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:25.874497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:57651 (local address 127.22.157.254:41669)
0928 15:51:25.874677 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:25.874681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:25.875376 (+ 695us) server_negotiation.cc:407] Connection header received
0928 15:51:25.875606 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:25.875610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:25.875673 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:25.875775 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:25.877274 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.878379 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.879767 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.880187 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.880324 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:25.882796 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:25.882801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:25.882804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:25.882836 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:25.885789 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:25.886782 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:25.886786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:25.886788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:25.886904 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:25.887380 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:25.887384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:25.887389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:25.887780 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:25.887947 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:25.888466 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 15:51:25.888647 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 15:51:25.889710 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:25.874691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:41669 (local address 127.22.157.194:57651)
0928 15:51:25.875272 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:25.875290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:25.875452 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:25.875803 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:25.875808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:25.876570 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:25.877121 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:25.877141 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.878524 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.878530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:25.879622 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:25.879634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:25.880369 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:25.880375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:25.880551 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:25.881239 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:25.881268 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:25.882558 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:25.885917 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:25.885925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:25.885932 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:25.886654 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:25.887008 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:25.887015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:25.887021 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:25.887251 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:25.887909 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:25.887915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:25.888185 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 15:51:25.888378 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 15:51:25.888897 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":324,"threads_started":1}
I20260928 15:51:25.891970 31066 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:42245
I20260928 15:51:25.891976 31202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:42245 every 8 connection(s)
I20260928 15:51:25.893499 31066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:51:25.895756 31203 heartbeater.cc:347] Connected to a master server at 127.22.157.254:41669
I20260928 15:51:25.895946 31203 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:25.896593 31203 heartbeater.cc:511] Master 127.22.157.254:41669 requested a full tablet report, sending...
I20260928 15:51:25.897658 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 31066
I20260928 15:51:25.897825 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:51:25.898049 30854 ts_manager.cc:194] Registered new tserver with Master: bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245)
I20260928 15:51:25.898285 23159 external_mini_cluster.cc:1428] Setting key d411ce697bd94095ff3250abb65add08
I20260928 15:51:25.899029 30854 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:39465
WARNING: no policy specified for oryx/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab.
Entry for principal oryx/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab.
I20260928 15:51:25.951965 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:41669
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab
--principal=oryx/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:26.302027 31207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:26.302373 31207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:26.302500 31207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:26.310227 31207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:26.310344 31207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:26.310436 31207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 15:51:26.310518 31207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:51:26.320817 31207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab
--principal=oryx/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.157.254:41669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:26.322358 31207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:26.323768 31207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:26.339147 31215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:26.339221 31213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:26.339178 31212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:26.340065 31207 server_base.cc:1034] running on GCE node
I20260928 15:51:26.340304 31062 heartbeater.cc:503] Master 127.22.157.254:41669 was elected leader, sending a full tablet report...
I20260928 15:51:26.340840 31207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:26.341893 31207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:26.343185 31207 hybrid_clock.cc:648] HybridClock initialized: now 1790610686343108 us; error 44 us; skew 500 ppm
I20260928 15:51:26.343463 31207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:26 dist-test-slave-1vw0 krb5kdc[30808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610686, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:26.346578 31207 init.cc:377] Logged in from keytab as oryx/127.22.157.195@KRBTEST.COM (short username oryx)
I20260928 15:51:26.348098 31207 webserver.cc:460] Webserver started at http://127.22.157.195:33535/ using document root <none> and password file <none>
I20260928 15:51:26.348848 31207 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:26.348969 31207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:26.349295 31207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:26.352169 31207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "00b0bacdf4a241c0b8f24ec497efe0c7"
format_stamp: "Formatted at 2026-09-28 15:51:26 on dist-test-slave-1vw0"
server_key: "f998efb6d2ef09b09f7ff07ac0e89fca"
server_key_iv: "3a2d46349ea566a7c8a4c33a3b4417a2"
server_key_version: "encryptionkey@0"
I20260928 15:51:26.353031 31207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "00b0bacdf4a241c0b8f24ec497efe0c7"
format_stamp: "Formatted at 2026-09-28 15:51:26 on dist-test-slave-1vw0"
server_key: "f998efb6d2ef09b09f7ff07ac0e89fca"
server_key_iv: "3a2d46349ea566a7c8a4c33a3b4417a2"
server_key_version: "encryptionkey@0"
I20260928 15:51:26.359375 31207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:51:26.364188 31222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:26.366110 31207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:51:26.366307 31207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "00b0bacdf4a241c0b8f24ec497efe0c7"
format_stamp: "Formatted at 2026-09-28 15:51:26 on dist-test-slave-1vw0"
server_key: "f998efb6d2ef09b09f7ff07ac0e89fca"
server_key_iv: "3a2d46349ea566a7c8a4c33a3b4417a2"
server_key_version: "encryptionkey@0"
I20260928 15:51:26.366587 31207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:26.384619 31207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:26.388639 31207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:26.388993 31207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:26.391614 31207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:26.391690 31207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:26.391825 31207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:26.391901 31207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:51:26 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610686, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:26.420097 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.392665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:54871 (local address 127.22.157.254:41669)
0928 15:51:26.393262 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.393274 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.394623 (+ 1349us) server_negotiation.cc:407] Connection header received
0928 15:51:26.397724 (+ 3101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.397734 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.397862 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.398017 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:26.400888 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.402033 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.404755 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.405590 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.405760 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.411548 (+ 5788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:26.411554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:26.411557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:26.411596 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:26.415729 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:26.417104 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:26.417111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:26.417116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:26.417317 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:26.417825 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:26.417833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:26.417839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:26.418309 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:26.418514 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:26.419280 (+ 766us) server_negotiation.cc:299] Negotiation successful
0928 15:51:26.419605 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":294,"threads_started":1}
I20260928 15:51:26.421648 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.393419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:41669 (local address 127.22.157.195:54871)
0928 15:51:26.394441 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:26.394493 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:26.397298 (+ 2805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:26.398462 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:26.398473 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:26.399490 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:26.400531 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.400568 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.402206 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.402213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.404423 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.404441 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.405915 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.405926 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.406329 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.407306 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:26.407348 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:26.411216 (+ 3868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:26.415967 (+ 4751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:26.415982 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:26.416013 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:26.416899 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:26.417448 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:26.417458 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:26.417460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:26.417688 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:26.418442 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:26.418449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:26.418870 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 15:51:26.419110 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 15:51:26.419630 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":425,"threads_started":1}
I20260928 15:51:26.439988 31207 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:45679
I20260928 15:51:26.439997 31343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:45679 every 8 connection(s)
I20260928 15:51:26.441612 31207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:51:26.443977 31344 heartbeater.cc:347] Connected to a master server at 127.22.157.254:41669
I20260928 15:51:26.444235 31344 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:26.445077 31344 heartbeater.cc:511] Master 127.22.157.254:41669 requested a full tablet report, sending...
I20260928 15:51:26.446545 30853 ts_manager.cc:194] Registered new tserver with Master: 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679)
I20260928 15:51:26.446740 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.430254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:52653 (local address 127.22.157.254:41669)
0928 15:51:26.430417 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.430421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.431080 (+ 659us) server_negotiation.cc:407] Connection header received
0928 15:51:26.431344 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.431350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.431466 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.431601 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:26.433061 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.434403 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.436854 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.437628 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.437837 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.440446 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:26.440450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:26.440454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:26.440486 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:26.443829 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:26.444680 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:26.444688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:26.444692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:26.444866 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:26.445267 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:26.445272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:26.445275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:26.445549 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:26.445710 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:26.446115 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 15:51:26.446369 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:51:26.446823 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.430363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:41669 (local address 127.22.157.195:52653)
0928 15:51:26.430975 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:26.430992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:26.431162 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:26.431648 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:26.431656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:26.432486 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:26.432881 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.432898 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.434624 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.434633 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.436625 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.436639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.438064 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.438069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.438182 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.438909 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:26.438936 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:26.440239 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:26.443987 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:26.443993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:26.443997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:26.444557 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:26.444986 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:26.444991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:26.444995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:26.445131 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:26.445674 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:26.445680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:26.445869 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 15:51:26.446009 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:51:26.446340 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":333,"threads_started":1}
I20260928 15:51:26.447713 30853 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:54871
I20260928 15:51:26.450382 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 31207
I20260928 15:51:26.450549 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:51:26.450986 23159 external_mini_cluster.cc:1428] Setting key d3b2c59cf8c5239ab555da50eac2b5e0
I20260928 15:51:26.453987 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 15:51:26 dist-test-slave-1vw0 krb5kdc[30808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 15:51:26 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:26 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:51:26.508810 31231 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:26.497891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.22.157.254:41669)
0928 15:51:26.498126 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.498135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.498264 (+ 129us) server_negotiation.cc:407] Connection header received
0928 15:51:26.498437 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.498441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.498549 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.498642 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:26.500030 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.500813 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.502119 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.502527 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.502659 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.508515 (+ 5856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54512: BlockingRecv error: recv got EOF from 127.0.0.1:54512 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 15:51:26 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610686, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:26.526396 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.511662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.22.157.254:41669)
0928 15:51:26.511897 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.511903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.512178 (+ 275us) server_negotiation.cc:407] Connection header received
0928 15:51:26.512379 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.512384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.512501 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.512608 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:26.513766 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.514615 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.515980 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.516538 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.516695 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.520809 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:26.520816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:26.520819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:26.520856 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:26.523574 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:26.524345 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:26.524354 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:26.524357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:26.524477 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:26.524897 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:26.524902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:26.524906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:26.525228 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:26.525494 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:26.525852 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 15:51:26.526096 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:51:26.530510 30853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:51:26.538116 30853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:51:26.568213 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.553691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.22.157.194:42245)
0928 15:51:26.554982 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.554991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.555019 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:51:26.555112 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.555119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.555368 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.556861 (+ 1493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:26.557128 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.561187 (+ 4059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.565461 (+ 4274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.566842 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.567008 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.567148 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:26.567736 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 15:51:26.567933 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1146,"thread_start_us":1027,"threads_started":1}
I20260928 15:51:26.568279 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.552560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:42245 (local address 127.0.0.1:34574)
0928 15:51:26.554265 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:26.554288 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:26.554425 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:26.555602 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:26.555606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:26.555632 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:26.556569 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.556586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.561412 (+ 4826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.561421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.565242 (+ 3821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.565258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.567319 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.567326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.567537 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.567562 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 15:51:26.567661 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 15:51:26.567812 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1467,"mutex_wait_us":1115,"thread_start_us":11790,"threads_started":1}
I20260928 15:51:26.568626 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.551881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:45679 (local address 127.0.0.1:40428)
0928 15:51:26.552677 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:26.552757 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:26.552956 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:26.554989 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:26.555002 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:26.555051 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:26.555782 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.555802 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.559263 (+ 3461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.559272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.563204 (+ 3932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.563230 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.566894 (+ 3664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.566901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.567141 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.567208 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 15:51:26.567369 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 15:51:26.567600 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":2,"thread_start_us":336,"threads_started":1}
I20260928 15:51:26.568823 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.552300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.22.157.195:45679)
0928 15:51:26.553966 (+ 1666us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.553980 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.554023 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:51:26.554180 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.554192 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.554563 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.556101 (+ 1538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:26.556506 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.559032 (+ 2526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.564953 (+ 5921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.566623 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.566905 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.567122 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:26.567947 (+ 825us) server_negotiation.cc:299] Negotiation successful
0928 15:51:26.568176 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1427,"thread_start_us":1217,"threads_started":1}
I20260928 15:51:26.572664 31133 tablet_service.cc:1467] Processing CreateTablet for tablet 448faa2c66e9432293f3dea51167b8cc (DEFAULT_TABLE table=test-table [id=8cd32dce574c4c0f9a19f14a1463ca28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:26.572664 31274 tablet_service.cc:1467] Processing CreateTablet for tablet 448faa2c66e9432293f3dea51167b8cc (DEFAULT_TABLE table=test-table [id=8cd32dce574c4c0f9a19f14a1463ca28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:26.574191 31133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448faa2c66e9432293f3dea51167b8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:26.574338 31274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448faa2c66e9432293f3dea51167b8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:26.575595 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.553291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:40665 (local address 127.0.0.1:57014)
0928 15:51:26.564771 (+ 11480us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:26.564798 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:26.564980 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:26.566130 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:26.566135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:26.566165 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:26.566551 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.566561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.570281 (+ 3720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.570287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.572467 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.572489 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.574607 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.574614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.574818 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.574863 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 15:51:26.575024 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 15:51:26.575215 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11259,"thread_start_us":1607,"threads_started":1}
I20260928 15:51:26.575647 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.554380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.22.157.193:40665)
0928 15:51:26.565534 (+ 11154us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.565550 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.565578 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:51:26.565663 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.565668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.565890 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.566167 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:26.566727 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.570096 (+ 3369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.572687 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.574293 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.574529 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.574728 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:26.575075 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 15:51:26.575290 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10998,"thread_start_us":10871,"threads_started":1}
I20260928 15:51:26.580238 30992 tablet_service.cc:1467] Processing CreateTablet for tablet 448faa2c66e9432293f3dea51167b8cc (DEFAULT_TABLE table=test-table [id=8cd32dce574c4c0f9a19f14a1463ca28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:26.581947 30992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 448faa2c66e9432293f3dea51167b8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:26.586748 31368 tablet_bootstrap.cc:492] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: Bootstrap starting.
I20260928 15:51:26.590442 31369 tablet_bootstrap.cc:492] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: Bootstrap starting.
I20260928 15:51:26.590807 31368 tablet_bootstrap.cc:654] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:26.592305 31368 log.cc:824] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:26.594252 31369 tablet_bootstrap.cc:654] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:26.595578 31369 log.cc:824] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:26.595837 31368 tablet_bootstrap.cc:492] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: No bootstrap required, opened a new log
I20260928 15:51:26.596128 31368 ts_tablet_manager.cc:1397] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 15:51:26.598333 31371 tablet_bootstrap.cc:492] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: Bootstrap starting.
I20260928 15:51:26.601609 31371 tablet_bootstrap.cc:654] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:26.602735 31371 log.cc:824] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:26.603636 31369 tablet_bootstrap.cc:492] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: No bootstrap required, opened a new log
I20260928 15:51:26.604004 31369 ts_tablet_manager.cc:1397] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260928 15:51:26.606695 31371 tablet_bootstrap.cc:492] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: No bootstrap required, opened a new log
I20260928 15:51:26.607024 31371 ts_tablet_manager.cc:1397] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260928 15:51:26.608188 31368 raft_consensus.cc:348] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } }
I20260928 15:51:26.608696 31368 raft_consensus.cc:374] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:26.608807 31368 raft_consensus.cc:729] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb8e4be66de4b3d9e22145f08d9824f, State: Initialized, Role: FOLLOWER
I20260928 15:51:26.609463 31368 consensus_queue.cc:260] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } }
I20260928 15:51:26.610634 31203 heartbeater.cc:503] Master 127.22.157.254:41669 was elected leader, sending a full tablet report...
I20260928 15:51:26.612833 31368 ts_tablet_manager.cc:1428] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260928 15:51:26.614367 31369 raft_consensus.cc:348] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } }
I20260928 15:51:26.614850 31369 raft_consensus.cc:374] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:26.614974 31369 raft_consensus.cc:729] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b0bacdf4a241c0b8f24ec497efe0c7, State: Initialized, Role: FOLLOWER
I20260928 15:51:26.615676 31369 consensus_queue.cc:260] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } }
I20260928 15:51:26.617166 31371 raft_consensus.cc:348] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } }
I20260928 15:51:26.617678 31371 raft_consensus.cc:374] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:26.617784 31371 raft_consensus.cc:729] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7fa319f5ab40069abf57f70b3bb7cb, State: Initialized, Role: FOLLOWER
I20260928 15:51:26.618345 31371 consensus_queue.cc:260] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } }
I20260928 15:51:26.619622 31344 heartbeater.cc:503] Master 127.22.157.254:41669 was elected leader, sending a full tablet report...
I20260928 15:51:26.620330 31369 ts_tablet_manager.cc:1428] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent starting tablet: real 0.016s user 0.008s sys 0.010s
I20260928 15:51:26.620771 31371 ts_tablet_manager.cc:1428] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
W20260928 15:51:26.645627 31204 tablet.cc:2369] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:51:26.693933 31345 tablet.cc:2369] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:51:26.749576 31374 raft_consensus.cc:482] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:26.749938 31374 raft_consensus.cc:504] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } }
I20260928 15:51:26.751693 31374 leader_election.cc:288] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679), 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665)
I20260928 15:51:26.758073 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.752070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:47903 (local address 127.22.157.195:45679)
0928 15:51:26.752334 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.752342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.752679 (+ 337us) server_negotiation.cc:407] Connection header received
0928 15:51:26.752933 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.752937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.753034 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.753138 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:26.753828 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.754576 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.756294 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.757135 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.757319 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.757404 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:26.757684 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 15:51:26.757814 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:51:26.758149 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.751956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:45679 (local address 127.22.157.194:47903)
0928 15:51:26.752554 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:26.752572 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:26.752773 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:26.753324 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:26.753329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:26.753361 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:26.753681 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.753689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.754705 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.754714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.756137 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.756149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.757367 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.757374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.757524 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.757548 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:51:26.757657 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:51:26.757794 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":307,"threads_started":1}
I20260928 15:51:26.759359 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.752948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:37995 (local address 127.22.157.193:40665)
0928 15:51:26.753229 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.753234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.753569 (+ 335us) server_negotiation.cc:407] Connection header received
0928 15:51:26.753775 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.753780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.753845 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.753938 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:26.754514 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.755224 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.757349 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.758174 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.758357 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.758421 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:26.758957 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 15:51:26.759109 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 15:51:26.759372 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.752802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:40665 (local address 127.22.157.194:37995)
0928 15:51:26.753461 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:26.753475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:26.753626 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:26.753956 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:26.753961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:26.753984 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:26.754350 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.754358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.755374 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.755379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.757166 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:26.757179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.758586 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.758594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:26.758777 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.758809 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:51:26.758907 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:51:26.759043 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":326,"threads_started":1}
I20260928 15:51:26.759253 31294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448faa2c66e9432293f3dea51167b8cc" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" is_pre_election: true
I20260928 15:51:26.759881 31294 raft_consensus.cc:2454] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 0.
I20260928 15:51:26.760257 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448faa2c66e9432293f3dea51167b8cc" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" is_pre_election: true
I20260928 15:51:26.760663 31082 leader_election.cc:302] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b0bacdf4a241c0b8f24ec497efe0c7, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:26.760854 31016 raft_consensus.cc:2454] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 0.
I20260928 15:51:26.761139 31374 raft_consensus.cc:2781] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:51:26.761324 31374 raft_consensus.cc:482] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:26.761442 31374 raft_consensus.cc:3037] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:26.764181 31374 raft_consensus.cc:504] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } }
I20260928 15:51:26.764814 31374 leader_election.cc:288] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 election: Requested vote from peers 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679), 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665)
I20260928 15:51:26.765280 31294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448faa2c66e9432293f3dea51167b8cc" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7"
I20260928 15:51:26.765328 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "448faa2c66e9432293f3dea51167b8cc" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb"
I20260928 15:51:26.765493 31294 raft_consensus.cc:3037] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:26.765530 31016 raft_consensus.cc:3037] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:26.767736 31016 raft_consensus.cc:2454] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 1.
I20260928 15:51:26.767737 31294 raft_consensus.cc:2454] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 1.
I20260928 15:51:26.768215 31082 leader_election.cc:302] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b0bacdf4a241c0b8f24ec497efe0c7, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:26.768512 31374 raft_consensus.cc:2781] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:26.769033 31374 raft_consensus.cc:686] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 1 LEADER]: Becoming Leader. State: Replica: bfb8e4be66de4b3d9e22145f08d9824f, State: Running, Role: LEADER
I20260928 15:51:26.769601 31374 consensus_queue.cc:237] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } }
I20260928 15:51:26.774641 30853 catalog_manager.cc:5495] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194). New cstate: current_term: 1 leader_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } health_report { overall_health: HEALTHY } } }
I20260928 15:51:26.819993 31366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:26.813834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.22.157.194:42245)
0928 15:51:26.814089 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:26.814096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:26.814247 (+ 151us) server_negotiation.cc:407] Connection header received
0928 15:51:26.814538 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:26.814544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:26.814652 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:26.814853 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:51:26.815196 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.816182 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.817738 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:26.818169 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:26.818287 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:26.818777 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:51:26.819375 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:51:26.819485 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:26.819589 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 15:51:26.819668 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:51:26.828630 31016 raft_consensus.cc:1261] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Refusing update from remote peer bfb8e4be66de4b3d9e22145f08d9824f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:26.828713 31294 raft_consensus.cc:1261] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Refusing update from remote peer bfb8e4be66de4b3d9e22145f08d9824f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:26.829679 31379 consensus_queue.cc:1035] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:26.830122 31374 consensus_queue.cc:1035] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:51:26.837334 31063 tablet.cc:2369] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:51:26.841567 31382 mvcc.cc:205] Tried to move back new op lower bound from 7334341373239742464 to 7334341373012647936. Current Snapshot: MvccSnapshot[applied={T|T < 7334341373239742464}]
I20260928 15:51:26.857375 30854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:26.857695 30854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:26.861948 30854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f: Sending DeleteTablet for 3 replicas of tablet 448faa2c66e9432293f3dea51167b8cc
I20260928 15:51:26.863446 30992 tablet_service.cc:1514] Processing DeleteTablet for tablet 448faa2c66e9432293f3dea51167b8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:26 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:57014
I20260928 15:51:26.863404 31133 tablet_service.cc:1514] Processing DeleteTablet for tablet 448faa2c66e9432293f3dea51167b8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:26 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:34574
I20260928 15:51:26.864480 31274 tablet_service.cc:1514] Processing DeleteTablet for tablet 448faa2c66e9432293f3dea51167b8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:26 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:40428
I20260928 15:51:26.865231 30854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:51:26.866006 30854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:51:26.875136 31393 tablet_replica.cc:330] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: stopping tablet replica
I20260928 15:51:26.875690 31393 raft_consensus.cc:2229] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:51:26.876989 30991 tablet_service.cc:1467] Processing CreateTablet for tablet af604e5bfc3b422f85f4b4bfcf322866 (DEFAULT_TABLE table=test-table [id=bb0b2f2bd9534cc3a84308ebcb956f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:26.877658 30991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af604e5bfc3b422f85f4b4bfcf322866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:26.877532 31132 tablet_service.cc:1467] Processing CreateTablet for tablet af604e5bfc3b422f85f4b4bfcf322866 (DEFAULT_TABLE table=test-table [id=bb0b2f2bd9534cc3a84308ebcb956f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:26.878527 31132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af604e5bfc3b422f85f4b4bfcf322866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:26.878839 31394 tablet_replica.cc:330] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: stopping tablet replica
I20260928 15:51:26.879391 31394 raft_consensus.cc:2229] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:26.879994 31394 raft_consensus.cc:2258] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:26.882434 31394 ts_tablet_manager.cc:1905] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:26.885540 31273 tablet_service.cc:1467] Processing CreateTablet for tablet af604e5bfc3b422f85f4b4bfcf322866 (DEFAULT_TABLE table=test-table [id=bb0b2f2bd9534cc3a84308ebcb956f3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:26.885679 31368 tablet_bootstrap.cc:492] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: Bootstrap starting.
I20260928 15:51:26.886041 31273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af604e5bfc3b422f85f4b4bfcf322866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:26.886710 31393 raft_consensus.cc:2258] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:26.887621 31368 tablet_bootstrap.cc:654] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:26.887784 31392 tablet_replica.cc:330] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: stopping tablet replica
I20260928 15:51:26.888289 31392 raft_consensus.cc:2229] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:26.888814 31392 raft_consensus.cc:2258] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:26.891600 31392 ts_tablet_manager.cc:1905] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:26.899941 31392 ts_tablet_manager.cc:1918] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:26.900174 31392 log.cc:1196] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/448faa2c66e9432293f3dea51167b8cc
I20260928 15:51:26.900851 31392 ts_tablet_manager.cc:1939] T 448faa2c66e9432293f3dea51167b8cc P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting consensus metadata
I20260928 15:51:26.901600 31394 ts_tablet_manager.cc:1918] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:26.902381 31394 log.cc:1196] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/448faa2c66e9432293f3dea51167b8cc
I20260928 15:51:26.903091 30837 catalog_manager.cc:4841] TS 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665): tablet 448faa2c66e9432293f3dea51167b8cc (table test-table [id=8cd32dce574c4c0f9a19f14a1463ca28]) successfully deleted
I20260928 15:51:26.904433 31368 tablet_bootstrap.cc:492] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: No bootstrap required, opened a new log
I20260928 15:51:26.904677 31368 ts_tablet_manager.cc:1397] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.005s
I20260928 15:51:26.905646 31368 raft_consensus.cc:348] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:26.905979 31368 raft_consensus.cc:374] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:26.906109 31368 raft_consensus.cc:729] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb8e4be66de4b3d9e22145f08d9824f, State: Initialized, Role: FOLLOWER
I20260928 15:51:26.906531 31369 tablet_bootstrap.cc:492] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: Bootstrap starting.
I20260928 15:51:26.906426 31368 consensus_queue.cc:260] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:26.907097 31368 ts_tablet_manager.cc:1428] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:51:26.908097 31394 ts_tablet_manager.cc:1939] T 448faa2c66e9432293f3dea51167b8cc P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting consensus metadata
I20260928 15:51:26.908334 31369 tablet_bootstrap.cc:654] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:26.909652 31371 tablet_bootstrap.cc:492] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: Bootstrap starting.
I20260928 15:51:26.911469 31371 tablet_bootstrap.cc:654] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:26.912243 31393 ts_tablet_manager.cc:1905] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:26.919001 30837 catalog_manager.cc:4841] TS 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679): tablet 448faa2c66e9432293f3dea51167b8cc (table test-table [id=8cd32dce574c4c0f9a19f14a1463ca28]) successfully deleted
I20260928 15:51:26.920694 31393 ts_tablet_manager.cc:1918] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:26.920995 31393 log.cc:1196] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/448faa2c66e9432293f3dea51167b8cc
I20260928 15:51:26.921731 31393 ts_tablet_manager.cc:1939] T 448faa2c66e9432293f3dea51167b8cc P bfb8e4be66de4b3d9e22145f08d9824f: Deleting consensus metadata
I20260928 15:51:26.923610 30840 catalog_manager.cc:4841] TS bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245): tablet 448faa2c66e9432293f3dea51167b8cc (table test-table [id=8cd32dce574c4c0f9a19f14a1463ca28]) successfully deleted
I20260928 15:51:26.927263 31369 tablet_bootstrap.cc:492] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: No bootstrap required, opened a new log
I20260928 15:51:26.927433 31371 tablet_bootstrap.cc:492] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: No bootstrap required, opened a new log
I20260928 15:51:26.927472 31369 ts_tablet_manager.cc:1397] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.003s
I20260928 15:51:26.927680 31371 ts_tablet_manager.cc:1397] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20260928 15:51:26.928300 31369 raft_consensus.cc:348] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:26.928519 31369 raft_consensus.cc:374] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:26.928611 31369 raft_consensus.cc:729] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b0bacdf4a241c0b8f24ec497efe0c7, State: Initialized, Role: FOLLOWER
I20260928 15:51:26.928525 31371 raft_consensus.cc:348] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:26.928824 31371 raft_consensus.cc:374] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:26.928932 31371 raft_consensus.cc:729] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7fa319f5ab40069abf57f70b3bb7cb, State: Initialized, Role: FOLLOWER
I20260928 15:51:26.928936 31369 consensus_queue.cc:260] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:26.929551 31369 ts_tablet_manager.cc:1428] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:51:26.929255 31371 consensus_queue.cc:260] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:26.931572 31371 ts_tablet_manager.cc:1428] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 15:51:27.049607 31374 raft_consensus.cc:482] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:27.049840 31374 raft_consensus.cc:504] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.050643 31374 leader_election.cc:288] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665), 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679)
I20260928 15:51:27.051314 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af604e5bfc3b422f85f4b4bfcf322866" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" is_pre_election: true
I20260928 15:51:27.051359 31294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af604e5bfc3b422f85f4b4bfcf322866" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" is_pre_election: true
I20260928 15:51:27.051616 31016 raft_consensus.cc:2454] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 0.
I20260928 15:51:27.051659 31294 raft_consensus.cc:2454] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 0.
I20260928 15:51:27.052114 31082 leader_election.cc:302] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7fa319f5ab40069abf57f70b3bb7cb, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:27.052413 31374 raft_consensus.cc:2781] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:51:27.052543 31374 raft_consensus.cc:482] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:27.052639 31374 raft_consensus.cc:3037] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:27.056008 31374 raft_consensus.cc:504] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.056842 31374 leader_election.cc:288] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 election: Requested vote from peers 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665), 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679)
I20260928 15:51:27.057582 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af604e5bfc3b422f85f4b4bfcf322866" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb"
I20260928 15:51:27.057583 31294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af604e5bfc3b422f85f4b4bfcf322866" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7"
I20260928 15:51:27.057835 31016 raft_consensus.cc:3037] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:27.057835 31294 raft_consensus.cc:3037] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:27.061048 31294 raft_consensus.cc:2454] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 1.
I20260928 15:51:27.061048 31016 raft_consensus.cc:2454] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 1.
I20260928 15:51:27.061619 31082 leader_election.cc:302] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7fa319f5ab40069abf57f70b3bb7cb, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:27.062016 31374 raft_consensus.cc:2781] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:27.062215 31374 raft_consensus.cc:686] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 1 LEADER]: Becoming Leader. State: Replica: bfb8e4be66de4b3d9e22145f08d9824f, State: Running, Role: LEADER
I20260928 15:51:27.062534 31374 consensus_queue.cc:237] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.065728 30852 catalog_manager.cc:5495] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194). New cstate: current_term: 1 leader_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } health_report { overall_health: UNKNOWN } } }
Sep 28 15:51:27 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610684, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.254@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:27.129396 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.107195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:41669 (local address 127.0.0.1:54516)
0928 15:51:27.107947 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:27.107972 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:27.108161 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:27.108520 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:27.108525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:27.109362 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:27.109577 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.109587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.110764 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.110771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.111750 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.111762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.112534 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.112540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.112759 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.119902 (+ 7143us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:27.119927 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:27.122885 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:27.126759 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:27.126780 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:27.126789 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:27.127355 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:27.127745 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:27.127752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:27.127755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:27.127900 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:27.128425 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:27.128434 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:27.128621 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 15:51:27.128816 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 15:51:27.129076 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":362,"threads_started":1}
I20260928 15:51:27.130157 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.107264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.22.157.254:41669)
0928 15:51:27.108247 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.108252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.108276 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:51:27.108326 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.108329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.108381 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.108492 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:27.109797 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.110636 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.111894 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.112316 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.112478 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.123133 (+ 10655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:27.123140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:27.123143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:27.123178 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:27.126594 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:27.127491 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:27.127497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:27.127501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:27.127637 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:27.128001 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:27.128006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:27.128009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:27.128294 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:27.128447 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:27.129734 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0928 15:51:27.129910 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":689,"threads_started":1}
I20260928 15:51:27.136955 30852 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:54516:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 15:51:27.138182 30852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:51:27.145397 30991 tablet_service.cc:1467] Processing CreateTablet for tablet 52cbea7872c14dc7a510d4126db38767 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd07e9df7f56445b915dab95533c007e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:51:27.145789 31132 tablet_service.cc:1467] Processing CreateTablet for tablet 52cbea7872c14dc7a510d4126db38767 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd07e9df7f56445b915dab95533c007e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:51:27.145784 31273 tablet_service.cc:1467] Processing CreateTablet for tablet 52cbea7872c14dc7a510d4126db38767 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd07e9df7f56445b915dab95533c007e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 15:51:27.146085 30991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52cbea7872c14dc7a510d4126db38767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:27.146297 31273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52cbea7872c14dc7a510d4126db38767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:27.146303 31132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52cbea7872c14dc7a510d4126db38767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:27.154132 31369 tablet_bootstrap.cc:492] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: Bootstrap starting.
I20260928 15:51:27.154902 31371 tablet_bootstrap.cc:492] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: Bootstrap starting.
I20260928 15:51:27.155944 31369 tablet_bootstrap.cc:654] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:27.156780 31371 tablet_bootstrap.cc:654] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:27.156808 31368 tablet_bootstrap.cc:492] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: Bootstrap starting.
I20260928 15:51:27.158710 31368 tablet_bootstrap.cc:654] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:27.167459 31368 tablet_bootstrap.cc:492] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: No bootstrap required, opened a new log
I20260928 15:51:27.167737 31368 ts_tablet_manager.cc:1397] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 15:51:27.168663 31369 tablet_bootstrap.cc:492] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: No bootstrap required, opened a new log
I20260928 15:51:27.168958 31369 ts_tablet_manager.cc:1397] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260928 15:51:27.168918 31368 raft_consensus.cc:348] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.169278 31368 raft_consensus.cc:374] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:27.169386 31368 raft_consensus.cc:729] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb8e4be66de4b3d9e22145f08d9824f, State: Initialized, Role: FOLLOWER
I20260928 15:51:27.169775 31368 consensus_queue.cc:260] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.170122 31369 raft_consensus.cc:348] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.170450 31369 raft_consensus.cc:374] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:27.170584 31369 raft_consensus.cc:729] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b0bacdf4a241c0b8f24ec497efe0c7, State: Initialized, Role: FOLLOWER
I20260928 15:51:27.170358 31374 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.170838 31374 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:51:27.171041 31369 consensus_queue.cc:260] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.171663 31375 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.171976 31375 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:51:27.173593 31369 ts_tablet_manager.cc:1428] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 15:51:27.174288 31371 tablet_bootstrap.cc:492] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: No bootstrap required, opened a new log
I20260928 15:51:27.174471 31371 ts_tablet_manager.cc:1397] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.014s
I20260928 15:51:27.175135 31368 ts_tablet_manager.cc:1428] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260928 15:51:27.175304 31371 raft_consensus.cc:348] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.175575 31371 raft_consensus.cc:374] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:27.175659 31371 raft_consensus.cc:729] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7fa319f5ab40069abf57f70b3bb7cb, State: Initialized, Role: FOLLOWER
I20260928 15:51:27.175956 31371 consensus_queue.cc:260] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.176657 31371 ts_tablet_manager.cc:1428] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 15:51:27.176496 31376 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.176968 31376 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:51:27.323978 31375 raft_consensus.cc:482] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:27.324259 31375 raft_consensus.cc:504] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.325860 31375 leader_election.cc:288] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665), bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245)
I20260928 15:51:27.332643 31376 raft_consensus.cc:482] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:27.332911 31376 raft_consensus.cc:504] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.334599 31376 leader_election.cc:288] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245), 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679)
I20260928 15:51:27.346097 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.326552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:32985 (local address 127.22.157.193:40665)
0928 15:51:27.337299 (+ 10747us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.337307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.337559 (+ 252us) server_negotiation.cc:407] Connection header received
0928 15:51:27.338390 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.338399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.338509 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.338635 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:27.339471 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.340891 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.343486 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.344817 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.344969 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.345048 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:27.345565 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 15:51:27.345717 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10633,"thread_start_us":18925,"threads_started":1}
I20260928 15:51:27.347200 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.326190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:42245 (local address 127.22.157.195:35267)
0928 15:51:27.338069 (+ 11879us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:27.338093 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:27.338229 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:27.340319 (+ 2090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:27.340325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:27.340352 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:27.340627 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.340638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.342560 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.342565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.344638 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.344655 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.346347 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.346353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.346520 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.346577 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 15:51:27.346647 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 15:51:27.346769 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11724,"mutex_wait_us":1,"thread_start_us":11107,"threads_started":1}
I20260928 15:51:27.348073 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.326119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:40665 (local address 127.22.157.195:32985)
0928 15:51:27.337277 (+ 11158us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:27.337303 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:27.337465 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:27.338916 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:27.338921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:27.338950 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:27.339242 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.339258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.341104 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.341111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.343288 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.343309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.345181 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.345187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.345357 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.345391 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:51:27.346613 (+ 1222us) client_negotiation.cc:240] Negotiation successful
0928 15:51:27.346770 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11009,"thread_start_us":18037,"threads_started":1}
I20260928 15:51:27.348305 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.326399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:35267 (local address 127.22.157.194:42245)
0928 15:51:27.339849 (+ 13450us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.339855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.339875 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:51:27.339956 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.339964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.340069 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.340822 (+ 753us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:27.341000 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.342395 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.344895 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.346148 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.347576 (+ 1428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.347661 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:27.347720 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 15:51:27.347861 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13286,"thread_start_us":13748,"threads_started":1}
I20260928 15:51:27.349222 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cbea7872c14dc7a510d4126db38767" candidate_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" is_pre_election: true
I20260928 15:51:27.349664 31153 raft_consensus.cc:2454] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b0bacdf4a241c0b8f24ec497efe0c7 in term 0.
I20260928 15:51:27.350410 31225 leader_election.cc:302] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b0bacdf4a241c0b8f24ec497efe0c7, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:27.350947 31375 raft_consensus.cc:2781] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:51:27.351073 31375 raft_consensus.cc:482] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:27.351187 31375 raft_consensus.cc:3037] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:27.351102 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cbea7872c14dc7a510d4126db38767" candidate_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" is_pre_election: true
I20260928 15:51:27.351455 31016 raft_consensus.cc:2454] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b0bacdf4a241c0b8f24ec497efe0c7 in term 0.
I20260928 15:51:27.354619 31375 raft_consensus.cc:504] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.355799 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cbea7872c14dc7a510d4126db38767" candidate_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb"
I20260928 15:51:27.355878 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cbea7872c14dc7a510d4126db38767" candidate_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb8e4be66de4b3d9e22145f08d9824f"
I20260928 15:51:27.356082 31153 raft_consensus.cc:3037] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:27.356086 31016 raft_consensus.cc:3037] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:27.359249 31153 raft_consensus.cc:2454] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b0bacdf4a241c0b8f24ec497efe0c7 in term 1.
I20260928 15:51:27.359357 31016 raft_consensus.cc:2454] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b0bacdf4a241c0b8f24ec497efe0c7 in term 1.
I20260928 15:51:27.359722 31225 leader_election.cc:302] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b0bacdf4a241c0b8f24ec497efe0c7, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:27.360116 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.336251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:41719 (local address 127.22.157.194:42245)
0928 15:51:27.348994 (+ 12743us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.349003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.349864 (+ 861us) server_negotiation.cc:407] Connection header received
0928 15:51:27.349929 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.349933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.350007 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.350103 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:27.352328 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.353455 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.357537 (+ 4082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.358845 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.358993 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.359081 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:27.359682 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 15:51:27.359835 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12638,"thread_start_us":12921,"threads_started":1}
I20260928 15:51:27.360184 31375 leader_election.cc:288] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [CANDIDATE]: Term 1 election: Requested vote from peers 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665), bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245)
I20260928 15:51:27.360549 31375 raft_consensus.cc:2781] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:27.360919 31375 raft_consensus.cc:686] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 LEADER]: Becoming Leader. State: Replica: 00b0bacdf4a241c0b8f24ec497efe0c7, State: Running, Role: LEADER
I20260928 15:51:27.360967 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.335664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:42245 (local address 127.22.157.193:41719)
0928 15:51:27.349593 (+ 13929us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:27.349612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:27.349752 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:27.351260 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:27.351266 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:27.351294 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:27.352099 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.352113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.353630 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.353635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.357332 (+ 3697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.357351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.359279 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.359287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.359472 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.359506 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:51:27.360363 (+ 857us) client_negotiation.cc:240] Negotiation successful
0928 15:51:27.360528 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13816,"mutex_wait_us":1,"thread_start_us":5236,"threads_started":1}
I20260928 15:51:27.361658 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cbea7872c14dc7a510d4126db38767" candidate_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" is_pre_election: true
I20260928 15:51:27.361583 31375 consensus_queue.cc:237] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.362109 31153 raft_consensus.cc:2379] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c7fa319f5ab40069abf57f70b3bb7cb in current term 1: Already voted for candidate 00b0bacdf4a241c0b8f24ec497efe0c7 in this term.
I20260928 15:51:27.362972 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.335068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:45679 (local address 127.22.157.193:46805)
0928 15:51:27.341184 (+ 6116us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:27.341208 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:27.341354 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:27.350984 (+ 9630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:27.350990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:27.351019 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:51:27.351502 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.351520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.354258 (+ 2738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.354264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.356517 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.356534 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.362092 (+ 5558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.362101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.362299 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.362332 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:51:27.362407 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 15:51:27.362564 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5974,"mutex_wait_us":113,"thread_start_us":14299,"threads_started":1}
I20260928 15:51:27.363744 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.344476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:46805 (local address 127.22.157.195:45679)
0928 15:51:27.350661 (+ 6185us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.350667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.350686 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:51:27.350768 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.350774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.350847 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.350967 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:51:27.352837 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.354070 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.360061 (+ 5991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.361820 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.363250 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.363333 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:27.363393 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 15:51:27.363496 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6029,"thread_start_us":5895,"threads_started":1}
I20260928 15:51:27.364331 31294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cbea7872c14dc7a510d4126db38767" candidate_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" is_pre_election: true
I20260928 15:51:27.365002 30941 leader_election.cc:302] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c7fa319f5ab40069abf57f70b3bb7cb; no voters: 00b0bacdf4a241c0b8f24ec497efe0c7, bfb8e4be66de4b3d9e22145f08d9824f
I20260928 15:51:27.365460 31376 raft_consensus.cc:2726] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 15:51:27.366097 31417 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: TxnStatusTablet state changed. Reason: New leader 00b0bacdf4a241c0b8f24ec497efe0c7. Latest consensus state: current_term: 1 leader_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.366457 31417 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:51:27.367045 31419 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 15:51:27.367209 31419 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 15:51:27.367157 30853 catalog_manager.cc:5495] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195). New cstate: current_term: 1 leader_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } health_report { overall_health: HEALTHY } } }
I20260928 15:51:27.415066 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.408302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.22.157.195:45679)
0928 15:51:27.408515 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.408524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.408626 (+ 102us) server_negotiation.cc:407] Connection header received
0928 15:51:27.408961 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.408965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.409082 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.409211 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:51:27.409620 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.410665 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.412640 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.413184 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.413287 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.413888 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:51:27.414492 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:51:27.414609 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:27.414741 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 15:51:27.414821 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 15:51:27.415230 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.408285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:45679 (local address 127.0.0.1:40436)
0928 15:51:27.408532 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:27.408553 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:27.408713 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:27.409260 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:27.409266 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:27.409280 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:51:27.409479 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.409492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.410809 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.410817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.412441 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.412457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.413348 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.413354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.413596 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.413677 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:51:27.414607 (+ 930us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:51:27.414634 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 15:51:27.414716 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:51:27.414889 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 15:51:27.420189 31016 raft_consensus.cc:1261] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Refusing update from remote peer 00b0bacdf4a241c0b8f24ec497efe0c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:27.420305 31153 raft_consensus.cc:1261] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Refusing update from remote peer 00b0bacdf4a241c0b8f24ec497efe0c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:27.421015 31375 consensus_queue.cc:1035] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:27.421326 31417 consensus_queue.cc:1035] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:27.433979 31376 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: TxnStatusTablet state changed. Reason: New leader 00b0bacdf4a241c0b8f24ec497efe0c7. Latest consensus state: current_term: 1 leader_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.434369 31376 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:51:27.436026 31374 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: TxnStatusTablet state changed. Reason: New leader 00b0bacdf4a241c0b8f24ec497efe0c7. Latest consensus state: current_term: 1 leader_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.436384 31374 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:51:27.437866 31423 mvcc.cc:205] Tried to move back new op lower bound from 7334341375665516544 to 7334341375444336640. Current Snapshot: MvccSnapshot[applied={T|T < 7334341375665516544}]
I20260928 15:51:27.441064 31376 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.441320 31376 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P 1c7fa319f5ab40069abf57f70b3bb7cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:51:27.451900 31417 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.452230 31417 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:51:27.455830 31417 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.456161 31417 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P 00b0bacdf4a241c0b8f24ec497efe0c7: This TxnStatusTablet replica's current role is: LEADER
I20260928 15:51:27.456163 31374 tablet_replica.cc:439] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } } }
I20260928 15:51:27.456447 31374 tablet_replica.cc:442] T 52cbea7872c14dc7a510d4126db38767 P bfb8e4be66de4b3d9e22145f08d9824f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 15:51:27.482633 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.469282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:50453 (local address 127.22.157.195:45679)
0928 15:51:27.469481 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.469490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.477491 (+ 8001us) server_negotiation.cc:407] Connection header received
0928 15:51:27.477746 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.477750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.477826 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.477926 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:51:27.478529 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.479341 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.480769 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.481236 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.481378 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.481849 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:51:27.481963 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:51:27.482059 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:27.482201 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 15:51:27.482289 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 15:51:27.482720 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.469300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:45679 (local address 127.22.157.194:50453)
0928 15:51:27.477347 (+ 8047us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:27.477370 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:27.477595 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:27.477945 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:27.477950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:27.477962 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:51:27.478303 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.478314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.479498 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.479504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.480564 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.480590 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.481434 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.481443 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.481604 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.481670 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:51:27.482061 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:51:27.482075 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:51:27.482175 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:51:27.482323 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7834,"thread_start_us":7673,"threads_started":1}
I20260928 15:51:27.494369 31294 raft_consensus.cc:1261] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Refusing update from remote peer bfb8e4be66de4b3d9e22145f08d9824f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:27.494441 31016 raft_consensus.cc:1261] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Refusing update from remote peer bfb8e4be66de4b3d9e22145f08d9824f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:27.494992 31379 consensus_queue.cc:1035] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:27.495285 31374 consensus_queue.cc:1035] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:27.537762 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.530620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:42071 (local address 127.22.157.194:42245)
0928 15:51:27.531015 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.531022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.531307 (+ 285us) server_negotiation.cc:407] Connection header received
0928 15:51:27.531586 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.531593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.531684 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.531775 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:51:27.532467 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.533395 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.535618 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.536293 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.536393 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.537115 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:51:27.537219 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:51:27.537292 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:27.537460 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 15:51:27.537532 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243}
I20260928 15:51:27.538040 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.530486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:42245 (local address 127.22.157.195:42071)
0928 15:51:27.531210 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:27.531230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:27.531436 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:27.531827 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:27.531833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:27.531848 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 15:51:27.532312 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.532326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.533533 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.533539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.535440 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:27.535457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.536597 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.536603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:27.536816 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.536892 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 15:51:27.537321 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 15:51:27.537335 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 15:51:27.537421 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:51:27.537598 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":337,"threads_started":1}
I20260928 15:51:27.573573 30852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:27.573913 30852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:27.577327 30852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f: Sending DeleteTablet for 3 replicas of tablet af604e5bfc3b422f85f4b4bfcf322866
I20260928 15:51:27.578262 31132 tablet_service.cc:1514] Processing DeleteTablet for tablet af604e5bfc3b422f85f4b4bfcf322866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:27 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:34574
I20260928 15:51:27.578413 31273 tablet_service.cc:1514] Processing DeleteTablet for tablet af604e5bfc3b422f85f4b4bfcf322866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:27 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:40428
I20260928 15:51:27.578874 30991 tablet_service.cc:1514] Processing DeleteTablet for tablet af604e5bfc3b422f85f4b4bfcf322866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:27 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:57014
I20260928 15:51:27.579660 31443 tablet_replica.cc:330] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: stopping tablet replica
I20260928 15:51:27.579962 31443 raft_consensus.cc:2229] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:27.580302 31443 raft_consensus.cc:2258] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:27.581559 31443 ts_tablet_manager.cc:1905] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:27.588877 31442 tablet_replica.cc:330] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: stopping tablet replica
I20260928 15:51:27.589537 31442 raft_consensus.cc:2229] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:27.589697 31443 ts_tablet_manager.cc:1918] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:51:27.589963 31443 log.cc:1196] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/af604e5bfc3b422f85f4b4bfcf322866
I20260928 15:51:27.590045 31442 raft_consensus.cc:2258] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:27.590538 31443 ts_tablet_manager.cc:1939] T af604e5bfc3b422f85f4b4bfcf322866 P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting consensus metadata
I20260928 15:51:27.591482 31442 ts_tablet_manager.cc:1905] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:27.591883 30837 catalog_manager.cc:4841] TS 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665): tablet af604e5bfc3b422f85f4b4bfcf322866 (table test-table [id=bb0b2f2bd9534cc3a84308ebcb956f3c]) successfully deleted
I20260928 15:51:27.597824 31441 tablet_replica.cc:330] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: stopping tablet replica
I20260928 15:51:27.598155 31441 raft_consensus.cc:2229] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:51:27.598701 31441 raft_consensus.cc:2258] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:27.599968 31441 ts_tablet_manager.cc:1905] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:27.601888 31442 ts_tablet_manager.cc:1918] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:51:27.602106 31442 log.cc:1196] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/af604e5bfc3b422f85f4b4bfcf322866
I20260928 15:51:27.602661 31442 ts_tablet_manager.cc:1939] T af604e5bfc3b422f85f4b4bfcf322866 P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting consensus metadata
I20260928 15:51:27.604045 30837 catalog_manager.cc:4841] TS 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679): tablet af604e5bfc3b422f85f4b4bfcf322866 (table test-table [id=bb0b2f2bd9534cc3a84308ebcb956f3c]) successfully deleted
I20260928 15:51:27.607717 31441 ts_tablet_manager.cc:1918] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:51:27.607865 31441 log.cc:1196] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/af604e5bfc3b422f85f4b4bfcf322866
I20260928 15:51:27.608455 31441 ts_tablet_manager.cc:1939] T af604e5bfc3b422f85f4b4bfcf322866 P bfb8e4be66de4b3d9e22145f08d9824f: Deleting consensus metadata
I20260928 15:51:27.609787 30840 catalog_manager.cc:4841] TS bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245): tablet af604e5bfc3b422f85f4b4bfcf322866 (table test-table [id=bb0b2f2bd9534cc3a84308ebcb956f3c]) successfully deleted
Sep 28 15:51:27 dist-test-slave-1vw0 krb5kdc[30808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 15:51:27 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:27 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:51:27.649339 31452 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:27.637451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.22.157.254:41669)
0928 15:51:27.638022 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.638029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.638324 (+ 295us) server_negotiation.cc:407] Connection header received
0928 15:51:27.638548 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.638554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.638657 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.638800 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:27.640267 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.641443 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.642744 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.643409 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.643542 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.649020 (+ 5478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54522: BlockingRecv error: recv got EOF from 127.0.0.1:54522 (error 108)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":242,"threads_started":1}
Sep 28 15:51:27 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM
I20260928 15:51:27.669399 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:27.652778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.22.157.254:41669)
0928 15:51:27.653002 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:27.653007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:27.653252 (+ 245us) server_negotiation.cc:407] Connection header received
0928 15:51:27.653430 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:27.653434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:27.653504 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:27.653600 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:27.655233 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.656013 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.657877 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:27.658314 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:27.658417 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:27.663513 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:27.663518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:27.663521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:27.663557 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:27.666265 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:27.667228 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:27.667233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:27.667235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:27.667383 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:27.667922 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:27.667926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:27.667928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:27.668245 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:27.668465 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:27.668937 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 15:51:27.669136 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 15:51:27.673386 30852 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:51:27.674103 30852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:51:27.681147 30991 tablet_service.cc:1467] Processing CreateTablet for tablet aa50c338f7c34eea88632b6be32ff82b (DEFAULT_TABLE table=test-table [id=602eb22874d144419b7f12b99971e970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:27.681493 31273 tablet_service.cc:1467] Processing CreateTablet for tablet aa50c338f7c34eea88632b6be32ff82b (DEFAULT_TABLE table=test-table [id=602eb22874d144419b7f12b99971e970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:27.681682 30991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa50c338f7c34eea88632b6be32ff82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:27.681979 31273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa50c338f7c34eea88632b6be32ff82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:27.685596 31132 tablet_service.cc:1467] Processing CreateTablet for tablet aa50c338f7c34eea88632b6be32ff82b (DEFAULT_TABLE table=test-table [id=602eb22874d144419b7f12b99971e970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:27.686033 31132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa50c338f7c34eea88632b6be32ff82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:27.701160 31460 tablet_bootstrap.cc:492] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: Bootstrap starting.
I20260928 15:51:27.703446 31459 tablet_bootstrap.cc:492] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: Bootstrap starting.
I20260928 15:51:27.705531 31460 tablet_bootstrap.cc:654] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:27.705835 31459 tablet_bootstrap.cc:654] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:27.708343 31458 tablet_bootstrap.cc:492] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: Bootstrap starting.
I20260928 15:51:27.710374 31458 tablet_bootstrap.cc:654] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:27.710477 31460 tablet_bootstrap.cc:492] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: No bootstrap required, opened a new log
I20260928 15:51:27.710682 31460 ts_tablet_manager.cc:1397] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 15:51:27.711722 31460 raft_consensus.cc:348] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.712091 31460 raft_consensus.cc:374] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:27.712249 31460 raft_consensus.cc:729] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb8e4be66de4b3d9e22145f08d9824f, State: Initialized, Role: FOLLOWER
I20260928 15:51:27.712607 31460 consensus_queue.cc:260] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.713433 31460 ts_tablet_manager.cc:1428] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 15:51:27.714856 31459 tablet_bootstrap.cc:492] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: No bootstrap required, opened a new log
I20260928 15:51:27.715034 31459 ts_tablet_manager.cc:1397] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20260928 15:51:27.716069 31459 raft_consensus.cc:348] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.716308 31459 raft_consensus.cc:374] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:27.716415 31459 raft_consensus.cc:729] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b0bacdf4a241c0b8f24ec497efe0c7, State: Initialized, Role: FOLLOWER
I20260928 15:51:27.716722 31459 consensus_queue.cc:260] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.718086 31459 ts_tablet_manager.cc:1428] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 15:51:27.718470 31458 tablet_bootstrap.cc:492] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: No bootstrap required, opened a new log
I20260928 15:51:27.718602 31458 ts_tablet_manager.cc:1397] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260928 15:51:27.719378 31458 raft_consensus.cc:348] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.719650 31458 raft_consensus.cc:374] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:27.719749 31458 raft_consensus.cc:729] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7fa319f5ab40069abf57f70b3bb7cb, State: Initialized, Role: FOLLOWER
I20260928 15:51:27.720021 31458 consensus_queue.cc:260] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.720688 31458 ts_tablet_manager.cc:1428] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:51:27.996863 31375 raft_consensus.cc:482] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:27.997090 31375 raft_consensus.cc:504] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:27.997732 31375 leader_election.cc:288] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665), bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245)
I20260928 15:51:27.998334 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa50c338f7c34eea88632b6be32ff82b" candidate_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" is_pre_election: true
I20260928 15:51:27.998343 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa50c338f7c34eea88632b6be32ff82b" candidate_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" is_pre_election: true
I20260928 15:51:27.998587 31153 raft_consensus.cc:2454] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b0bacdf4a241c0b8f24ec497efe0c7 in term 0.
I20260928 15:51:27.998593 31016 raft_consensus.cc:2454] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00b0bacdf4a241c0b8f24ec497efe0c7 in term 0.
I20260928 15:51:27.999092 31225 leader_election.cc:302] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b0bacdf4a241c0b8f24ec497efe0c7, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:27.999367 31375 raft_consensus.cc:2781] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:51:27.999485 31375 raft_consensus.cc:482] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:27.999624 31375 raft_consensus.cc:3037] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:28.001856 31375 raft_consensus.cc:504] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.002369 31375 leader_election.cc:288] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [CANDIDATE]: Term 1 election: Requested vote from peers 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665), bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245)
I20260928 15:51:28.002998 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa50c338f7c34eea88632b6be32ff82b" candidate_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb"
I20260928 15:51:28.002966 31153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa50c338f7c34eea88632b6be32ff82b" candidate_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb8e4be66de4b3d9e22145f08d9824f"
I20260928 15:51:28.003238 31153 raft_consensus.cc:3037] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:28.003242 31016 raft_consensus.cc:3037] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:28.006177 31153 raft_consensus.cc:2454] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b0bacdf4a241c0b8f24ec497efe0c7 in term 1.
I20260928 15:51:28.006178 31016 raft_consensus.cc:2454] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00b0bacdf4a241c0b8f24ec497efe0c7 in term 1.
I20260928 15:51:28.006606 31225 leader_election.cc:302] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b0bacdf4a241c0b8f24ec497efe0c7, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:28.006906 31375 raft_consensus.cc:2781] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:28.007053 31375 raft_consensus.cc:686] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 LEADER]: Becoming Leader. State: Replica: 00b0bacdf4a241c0b8f24ec497efe0c7, State: Running, Role: LEADER
I20260928 15:51:28.007305 31375 consensus_queue.cc:237] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.010047 30853 catalog_manager.cc:5495] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195). New cstate: current_term: 1 leader_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } health_report { overall_health: HEALTHY } } }
I20260928 15:51:28.025368 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:28.020096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.22.157.195:45679)
0928 15:51:28.020603 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:28.020609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:28.020629 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:51:28.020682 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:28.020686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:28.020807 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:28.020906 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:51:28.021291 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.022135 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.023499 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.023972 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.024103 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:28.024545 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:51:28.024698 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:51:28.024807 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:28.024962 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:51:28.025045 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":233,"threads_started":1}
I20260928 15:51:28.028590 31153 raft_consensus.cc:1261] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Refusing update from remote peer 00b0bacdf4a241c0b8f24ec497efe0c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:28.028600 31016 raft_consensus.cc:1261] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Refusing update from remote peer 00b0bacdf4a241c0b8f24ec497efe0c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:28.029229 31375 consensus_queue.cc:1035] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:28.029567 31417 consensus_queue.cc:1035] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:28.033690 31422 mvcc.cc:205] Tried to move back new op lower bound from 7334341378159026176 to 7334341378080157696. Current Snapshot: MvccSnapshot[applied={T|T < 7334341378159026176}]
I20260928 15:51:28.043929 30852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:28.044130 30852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:28.047225 30852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f: Sending DeleteTablet for 3 replicas of tablet aa50c338f7c34eea88632b6be32ff82b
I20260928 15:51:28.048134 30991 tablet_service.cc:1514] Processing DeleteTablet for tablet aa50c338f7c34eea88632b6be32ff82b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:28 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:57014
I20260928 15:51:28.048161 31132 tablet_service.cc:1514] Processing DeleteTablet for tablet aa50c338f7c34eea88632b6be32ff82b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:28 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:34574
I20260928 15:51:28.048445 31443 tablet_replica.cc:330] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: stopping tablet replica
I20260928 15:51:28.048655 31443 raft_consensus.cc:2229] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:28.048739 31441 tablet_replica.cc:330] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: stopping tablet replica
I20260928 15:51:28.048702 31273 tablet_service.cc:1514] Processing DeleteTablet for tablet aa50c338f7c34eea88632b6be32ff82b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:28 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:40428
I20260928 15:51:28.048913 31443 raft_consensus.cc:2258] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:28.048924 31441 raft_consensus.cc:2229] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:28.049168 31441 raft_consensus.cc:2258] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:28.049458 31442 tablet_replica.cc:330] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: stopping tablet replica
I20260928 15:51:28.049636 31442 raft_consensus.cc:2229] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:51:28.049999 31443 ts_tablet_manager.cc:1905] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:28.050060 31442 raft_consensus.cc:2258] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:28.051142 31441 ts_tablet_manager.cc:1905] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:28.054677 31443 ts_tablet_manager.cc:1918] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:28.054822 31443 log.cc:1196] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/aa50c338f7c34eea88632b6be32ff82b
I20260928 15:51:28.055282 31443 ts_tablet_manager.cc:1939] T aa50c338f7c34eea88632b6be32ff82b P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting consensus metadata
I20260928 15:51:28.055552 30852 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 15:51:28.056128 31442 ts_tablet_manager.cc:1905] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 15:51:28.056293 30852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:51:28.056408 30837 catalog_manager.cc:4841] TS 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665): tablet aa50c338f7c34eea88632b6be32ff82b (table test-table [id=602eb22874d144419b7f12b99971e970]) successfully deleted
I20260928 15:51:28.057809 31441 ts_tablet_manager.cc:1918] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:28.057969 31441 log.cc:1196] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/aa50c338f7c34eea88632b6be32ff82b
I20260928 15:51:28.058387 31441 ts_tablet_manager.cc:1939] T aa50c338f7c34eea88632b6be32ff82b P bfb8e4be66de4b3d9e22145f08d9824f: Deleting consensus metadata
I20260928 15:51:28.059473 30840 catalog_manager.cc:4841] TS bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245): tablet aa50c338f7c34eea88632b6be32ff82b (table test-table [id=602eb22874d144419b7f12b99971e970]) successfully deleted
I20260928 15:51:28.062973 31442 ts_tablet_manager.cc:1918] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:51:28.063148 31442 log.cc:1196] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/aa50c338f7c34eea88632b6be32ff82b
I20260928 15:51:28.063625 31442 ts_tablet_manager.cc:1939] T aa50c338f7c34eea88632b6be32ff82b P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting consensus metadata
I20260928 15:51:28.064774 30991 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6e7fd3a2864a19a48a9c21deaa59ca (DEFAULT_TABLE table=test-table [id=683cc6c1c1ea45228362dfb5f62d4910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:28.064908 30837 catalog_manager.cc:4841] TS 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679): tablet aa50c338f7c34eea88632b6be32ff82b (table test-table [id=602eb22874d144419b7f12b99971e970]) successfully deleted
I20260928 15:51:28.065217 30991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6e7fd3a2864a19a48a9c21deaa59ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:28.065235 31273 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6e7fd3a2864a19a48a9c21deaa59ca (DEFAULT_TABLE table=test-table [id=683cc6c1c1ea45228362dfb5f62d4910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:28.065549 31273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6e7fd3a2864a19a48a9c21deaa59ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:28.069517 31132 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6e7fd3a2864a19a48a9c21deaa59ca (DEFAULT_TABLE table=test-table [id=683cc6c1c1ea45228362dfb5f62d4910]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:51:28.069914 31132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6e7fd3a2864a19a48a9c21deaa59ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:28.070258 31459 tablet_bootstrap.cc:492] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: Bootstrap starting.
I20260928 15:51:28.071911 31458 tablet_bootstrap.cc:492] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: Bootstrap starting.
I20260928 15:51:28.071992 31459 tablet_bootstrap.cc:654] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:28.073571 31458 tablet_bootstrap.cc:654] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:28.078948 31460 tablet_bootstrap.cc:492] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: Bootstrap starting.
I20260928 15:51:28.080782 31460 tablet_bootstrap.cc:654] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:28.082312 31459 tablet_bootstrap.cc:492] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: No bootstrap required, opened a new log
I20260928 15:51:28.082542 31459 ts_tablet_manager.cc:1397] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20260928 15:51:28.084277 31458 tablet_bootstrap.cc:492] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: No bootstrap required, opened a new log
I20260928 15:51:28.084533 31458 ts_tablet_manager.cc:1397] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260928 15:51:28.085248 31459 raft_consensus.cc:348] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.085649 31459 raft_consensus.cc:374] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:28.085462 31458 raft_consensus.cc:348] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.085763 31459 raft_consensus.cc:729] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b0bacdf4a241c0b8f24ec497efe0c7, State: Initialized, Role: FOLLOWER
I20260928 15:51:28.085949 31458 raft_consensus.cc:374] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:28.086086 31458 raft_consensus.cc:729] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7fa319f5ab40069abf57f70b3bb7cb, State: Initialized, Role: FOLLOWER
I20260928 15:51:28.086144 31459 consensus_queue.cc:260] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.086484 31458 consensus_queue.cc:260] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.089648 31459 ts_tablet_manager.cc:1428] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260928 15:51:28.091308 31458 ts_tablet_manager.cc:1428] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260928 15:51:28.094890 31460 tablet_bootstrap.cc:492] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: No bootstrap required, opened a new log
I20260928 15:51:28.095086 31460 ts_tablet_manager.cc:1397] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260928 15:51:28.095757 31460 raft_consensus.cc:348] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.095963 31460 raft_consensus.cc:374] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:28.096058 31460 raft_consensus.cc:729] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb8e4be66de4b3d9e22145f08d9824f, State: Initialized, Role: FOLLOWER
I20260928 15:51:28.096320 31460 consensus_queue.cc:260] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.096859 31460 ts_tablet_manager.cc:1428] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 15:51:28.289387 31379 raft_consensus.cc:482] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:51:28.289659 31379 raft_consensus.cc:504] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.290524 31379 leader_election.cc:288] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665), 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679)
I20260928 15:51:28.291325 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6e7fd3a2864a19a48a9c21deaa59ca" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" is_pre_election: true
I20260928 15:51:28.291455 31294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6e7fd3a2864a19a48a9c21deaa59ca" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" is_pre_election: true
I20260928 15:51:28.291607 31016 raft_consensus.cc:2454] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 0.
I20260928 15:51:28.291778 31294 raft_consensus.cc:2454] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 0.
I20260928 15:51:28.292160 31082 leader_election.cc:302] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7fa319f5ab40069abf57f70b3bb7cb, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:28.292582 31379 raft_consensus.cc:2781] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:51:28.292709 31379 raft_consensus.cc:482] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:51:28.292835 31379 raft_consensus.cc:3037] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:28.296058 31379 raft_consensus.cc:504] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.296937 31379 leader_election.cc:288] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 election: Requested vote from peers 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665), 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679)
I20260928 15:51:28.297576 31016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6e7fd3a2864a19a48a9c21deaa59ca" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb"
I20260928 15:51:28.297679 31294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6e7fd3a2864a19a48a9c21deaa59ca" candidate_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7"
I20260928 15:51:28.297853 31016 raft_consensus.cc:3037] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:28.298017 31294 raft_consensus.cc:3037] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:28.300347 31294 raft_consensus.cc:2454] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 1.
I20260928 15:51:28.300909 31082 leader_election.cc:302] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00b0bacdf4a241c0b8f24ec497efe0c7, bfb8e4be66de4b3d9e22145f08d9824f; no voters:
I20260928 15:51:28.301028 31016 raft_consensus.cc:2454] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb8e4be66de4b3d9e22145f08d9824f in term 1.
I20260928 15:51:28.301288 31379 raft_consensus.cc:2781] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:28.301832 31379 raft_consensus.cc:686] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 1 LEADER]: Becoming Leader. State: Replica: bfb8e4be66de4b3d9e22145f08d9824f, State: Running, Role: LEADER
I20260928 15:51:28.302165 31379 consensus_queue.cc:237] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } }
I20260928 15:51:28.305333 30852 catalog_manager.cc:5495] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194). New cstate: current_term: 1 leader_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfb8e4be66de4b3d9e22145f08d9824f" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 42245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 } health_report { overall_health: UNKNOWN } } }
I20260928 15:51:28.350832 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:28.335798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.22.157.194:42245)
0928 15:51:28.345290 (+ 9492us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:28.345295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:28.345318 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:51:28.345391 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:28.345395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:28.345481 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:28.345633 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:51:28.346128 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.347041 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.348613 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.349232 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.349422 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:28.349892 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:51:28.350003 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:51:28.350101 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:28.350250 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 15:51:28.350336 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9370,"thread_start_us":9190,"threads_started":1}
I20260928 15:51:28.361936 31016 raft_consensus.cc:1261] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Refusing update from remote peer bfb8e4be66de4b3d9e22145f08d9824f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:28.361877 31294 raft_consensus.cc:1261] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Refusing update from remote peer bfb8e4be66de4b3d9e22145f08d9824f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:51:28.362602 31474 consensus_queue.cc:1035] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b0bacdf4a241c0b8f24ec497efe0c7" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:28.363219 31379 consensus_queue.cc:1035] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7fa319f5ab40069abf57f70b3bb7cb" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 40665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:51:28.429139 30853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:28.429388 30853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:51:28.432590 30853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd8001fd5dbb4c7d85b99072bf3ecf9f: Sending DeleteTablet for 3 replicas of tablet 8b6e7fd3a2864a19a48a9c21deaa59ca
I20260928 15:51:28.433504 30991 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b6e7fd3a2864a19a48a9c21deaa59ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:28 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:57014
I20260928 15:51:28.433836 31443 tablet_replica.cc:330] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: stopping tablet replica
I20260928 15:51:28.434042 31443 raft_consensus.cc:2229] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:28.434290 31443 raft_consensus.cc:2258] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:28.434556 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 30925
I20260928 15:51:28.435706 31443 ts_tablet_manager.cc:1905] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:28.435834 31132 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b6e7fd3a2864a19a48a9c21deaa59ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:28 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:34574
I20260928 15:51:28.436180 31441 tablet_replica.cc:330] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: stopping tablet replica
I20260928 15:51:28.436385 31441 raft_consensus.cc:2229] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:51:28.436771 31441 raft_consensus.cc:2258] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:28.437657 31273 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b6e7fd3a2864a19a48a9c21deaa59ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:51:28 UTC) from {username='oryx', principal='oryx/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:40428
I20260928 15:51:28.438025 31442 tablet_replica.cc:330] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: stopping tablet replica
I20260928 15:51:28.438253 31442 raft_consensus.cc:2229] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:51:28.438494 31442 raft_consensus.cc:2258] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:51:28.440068 31442 ts_tablet_manager.cc:1905] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:28.443317 31443 ts_tablet_manager.cc:1918] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:51:28.443459 31443 log.cc:1196] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/8b6e7fd3a2864a19a48a9c21deaa59ca
I20260928 15:51:28.443903 31443 ts_tablet_manager.cc:1939] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 1c7fa319f5ab40069abf57f70b3bb7cb: Deleting consensus metadata
I20260928 15:51:28.446309 31441 ts_tablet_manager.cc:1905] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:51:28.446372 30837 catalog_manager.cc:4841] TS 1c7fa319f5ab40069abf57f70b3bb7cb (127.22.157.193:40665): tablet 8b6e7fd3a2864a19a48a9c21deaa59ca (table test-table [id=683cc6c1c1ea45228362dfb5f62d4910]) successfully deleted
I20260928 15:51:28.451294 31442 ts_tablet_manager.cc:1918] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:51:28.451601 31442 log.cc:1196] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/8b6e7fd3a2864a19a48a9c21deaa59ca
I20260928 15:51:28.452224 31442 ts_tablet_manager.cc:1939] T 8b6e7fd3a2864a19a48a9c21deaa59ca P 00b0bacdf4a241c0b8f24ec497efe0c7: Deleting consensus metadata
I20260928 15:51:28.453794 30837 catalog_manager.cc:4841] TS 00b0bacdf4a241c0b8f24ec497efe0c7 (127.22.157.195:45679): tablet 8b6e7fd3a2864a19a48a9c21deaa59ca (table test-table [id=683cc6c1c1ea45228362dfb5f62d4910]) successfully deleted
I20260928 15:51:28.454223 31441 ts_tablet_manager.cc:1918] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 15:51:28.454438 31441 log.cc:1196] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/8b6e7fd3a2864a19a48a9c21deaa59ca
I20260928 15:51:28.454948 31441 ts_tablet_manager.cc:1939] T 8b6e7fd3a2864a19a48a9c21deaa59ca P bfb8e4be66de4b3d9e22145f08d9824f: Deleting consensus metadata
I20260928 15:51:28.456272 30840 catalog_manager.cc:4841] TS bfb8e4be66de4b3d9e22145f08d9824f (127.22.157.194:42245): tablet 8b6e7fd3a2864a19a48a9c21deaa59ca (table test-table [id=683cc6c1c1ea45228362dfb5f62d4910]) successfully deleted
Sep 28 15:51:28 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.193@KRBTEST.COM
I20260928 15:51:28.474684 31483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:28.437199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.22.157.193:40665)
0928 15:51:28.458359 (+ 21160us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:28.458364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:28.458384 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:51:28.458439 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:28.458442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:28.458521 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:28.458667 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:28.460096 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.460919 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.462195 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.462646 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.462785 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:28.467679 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:28.467697 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:28.467714 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:28.467760 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:28.471149 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:28.472200 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:28.472210 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:28.472212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:28.472374 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:28.472917 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:28.472924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:28.472928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:28.473310 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:28.473846 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:28.474016 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 15:51:28.474278 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21044,"thread_start_us":21598,"threads_started":1}
I20260928 15:51:28.475454 31057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:28.581598 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 30925
I20260928 15:51:28.602083 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 31066
Sep 28 15:51:28 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.194@KRBTEST.COM
I20260928 15:51:28.618690 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:28.602960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.22.157.194:42245)
0928 15:51:28.603160 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:28.603164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:28.603206 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:51:28.603391 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:28.603396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:28.603478 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:28.603572 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:28.604869 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.605662 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.607015 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.607458 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.607755 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:28.612279 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:28.612296 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:28.612313 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:28.612355 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:28.615758 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:28.616554 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:28.616561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:28.616564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:28.616914 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:28.617262 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:28.617266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:28.617268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:28.617664 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:28.617905 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:28.618129 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 15:51:28.618314 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:51:28.619401 31198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:28.753552 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 31066
I20260928 15:51:28.778477 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 31207
Sep 28 15:51:28 dist-test-slave-1vw0 krb5kdc[30808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.157.195@KRBTEST.COM
I20260928 15:51:28.797493 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:28.779275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.22.157.195:45679)
0928 15:51:28.779882 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:28.779887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:28.779904 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:51:28.779959 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:28.779963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:28.780057 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:28.780153 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:28.781587 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.782292 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.783452 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:28.783859 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:28.783955 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:28.790565 (+ 6610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:28.790589 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:28.790611 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:28.790670 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:28.794689 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:28.795474 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:28.795482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:28.795484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:28.795624 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:28.796053 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:28.796056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:28.796058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:28.796409 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:28.796650 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:28.797033 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 15:51:28.797232 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":339,"threads_started":1}
I20260928 15:51:28.798153 31339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:28.943141 31488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:28.942132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:42245 (local address 127.22.157.195:48241)
0928 15:51:28.942726 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:28.942902 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.194:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"threads_started":1}
I20260928 15:51:28.943156 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 31207
I20260928 15:51:28.969324 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 30821
I20260928 15:51:28.970561 30894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:29.087373 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 30821
2026-09-28T15:51:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6294 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:51:29 dist-test-slave-1vw0 krb5kdc[31493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:51:29 dist-test-slave-1vw0 krb5kdc[31493](info): set up 2 sockets
Sep 28 15:51:29 dist-test-slave-1vw0 krb5kdc[31493](info): commencing operation
krb5kdc: starting...
W20260928 15:51:30.214143 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:51:30 dist-test-slave-1vw0 krb5kdc[31493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:51:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:51:30.469900 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42143
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:42143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:30.807128 31506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:30.807511 31506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:30.807689 31506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:30.815191 31506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:30.815361 31506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:30.815487 31506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:30.815598 31506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:30.815692 31506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:30.825877 31506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:42143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:30.827291 31506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:30.828675 31506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:30.841662 31511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:30.841634 31514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:30.841856 31512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:30.842691 31506 server_base.cc:1034] running on GCE node
I20260928 15:51:30.843576 31506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:30.845170 31506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:30.846388 31506 hybrid_clock.cc:648] HybridClock initialized: now 1790610690846341 us; error 90 us; skew 500 ppm
I20260928 15:51:30.846735 31506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T15:51:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1832 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:51:31 dist-test-slave-1vw0 krb5kdc[31520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:51:31 dist-test-slave-1vw0 krb5kdc[31520](info): set up 2 sockets
Sep 28 15:51:31 dist-test-slave-1vw0 krb5kdc[31520](info): commencing operation
krb5kdc: starting...
W20260928 15:51:32.081985 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:51:32 dist-test-slave-1vw0 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:51:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:51:32.340709 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43797
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:41311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:43797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:32.689052 31533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:32.689436 31533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:32.689604 31533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:32.696983 31533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:32.697150 31533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:32.697263 31533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:32.697367 31533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:32.697444 31533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:32.707677 31533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:41311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:43797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:32.709303 31533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:32.710700 31533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:32.723803 31541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:32.723832 31538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:32.723985 31539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:32.724824 31533 server_base.cc:1034] running on GCE node
I20260928 15:51:32.725668 31533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:32.727157 31533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:32.728425 31533 hybrid_clock.cc:648] HybridClock initialized: now 1790610692728386 us; error 46 us; skew 500 ppm
I20260928 15:51:32.728899 31533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:32 dist-test-slave-1vw0 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610692, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:32.739982 31533 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:51:32.741815 31533 webserver.cc:460] Webserver started at http://127.22.157.254:36909/ using document root <none> and password file <none>
I20260928 15:51:32.742555 31533 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:32.742661 31533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:32.742933 31533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:32.746126 31533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "5e41847d15dd4410b4ffc33b153e232f"
format_stamp: "Formatted at 2026-09-28 15:51:32 on dist-test-slave-1vw0"
server_key: "529e3b6f4cbff12822962cfda74b0713"
server_key_iv: "4db362ecb33f247dbe3e92e26b9cb031"
server_key_version: "encryptionkey@0"
I20260928 15:51:32.746889 31533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "5e41847d15dd4410b4ffc33b153e232f"
format_stamp: "Formatted at 2026-09-28 15:51:32 on dist-test-slave-1vw0"
server_key: "529e3b6f4cbff12822962cfda74b0713"
server_key_iv: "4db362ecb33f247dbe3e92e26b9cb031"
server_key_version: "encryptionkey@0"
I20260928 15:51:32.752569 31533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:51:32.756488 31549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:32.758059 31533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:32.758216 31533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "5e41847d15dd4410b4ffc33b153e232f"
format_stamp: "Formatted at 2026-09-28 15:51:32 on dist-test-slave-1vw0"
server_key: "529e3b6f4cbff12822962cfda74b0713"
server_key_iv: "4db362ecb33f247dbe3e92e26b9cb031"
server_key_version: "encryptionkey@0"
I20260928 15:51:32.758396 31533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:32.775415 31533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:32.779982 31533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:32.780225 31533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:32.799845 31533 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:43797
I20260928 15:51:32.799858 31600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:43797 every 8 connection(s)
I20260928 15:51:32.801352 31533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:32.805317 31601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:32.806802 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 31533
I20260928 15:51:32.807022 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:32.807881 23159 external_mini_cluster.cc:1428] Setting key 78b411456695db0208bc06d78d612d39
I20260928 15:51:32.821062 31601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: Bootstrap starting.
Sep 28 15:51:32 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:51:32.824875 31601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:32.826148 31601 log.cc:824] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:32.829162 31601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: No bootstrap required, opened a new log
I20260928 15:51:32.831797 31604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:32.810055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.22.157.254:43797)
0928 15:51:32.810929 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:32.810942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:32.810990 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:51:32.812653 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:32.812678 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:32.813209 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:32.813573 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:32.815541 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:32.816975 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:32.818859 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:32.819559 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:32.819705 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:32.824302 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:32.824310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:32.824326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:32.824365 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:32.827593 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:32.828289 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:32.828297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:32.828304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:32.828470 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:32.828901 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:32.828907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:32.828911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:32.829495 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:32.829849 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:32.830283 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 15:51:32.830733 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":432,"threads_started":1}
I20260928 15:51:32.837311 31601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } }
I20260928 15:51:32.837697 31601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:32.837800 31601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e41847d15dd4410b4ffc33b153e232f, State: Initialized, Role: FOLLOWER
I20260928 15:51:32.838410 31601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } }
I20260928 15:51:32.838660 31601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:32.838800 31601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:32.838979 31601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:32.841609 31601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } }
I20260928 15:51:32.842087 31601 leader_election.cc:302] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e41847d15dd4410b4ffc33b153e232f; no voters:
I20260928 15:51:32.842628 31601 leader_election.cc:288] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:32.842788 31606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:32.843544 31606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 1 LEADER]: Becoming Leader. State: Replica: 5e41847d15dd4410b4ffc33b153e232f, State: Running, Role: LEADER
I20260928 15:51:32.844187 31606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } }
I20260928 15:51:32.845000 31601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:32.847683 31607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e41847d15dd4410b4ffc33b153e232f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } } }
I20260928 15:51:32.848124 31607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:32.848346 31608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e41847d15dd4410b4ffc33b153e232f. Latest consensus state: current_term: 1 leader_uuid: "5e41847d15dd4410b4ffc33b153e232f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } } }
I20260928 15:51:32.848694 31608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:32.852370 31614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:32.857935 31614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:32.867547 31614 catalog_manager.cc:1347] Generated new cluster ID: 5d790752e60f4c25b8c5ca356645534d
I20260928 15:51:32.867690 31614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:32.885831 31614 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:32.887517 31614 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:32.897984 31614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: Generated new TSK 0
I20260928 15:51:32.898890 31614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:51:32.973078 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43797
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:41311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:33.324610 31628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:33.324957 31628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:33.325057 31628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:33.333339 31628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:33.333460 31628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:33.333636 31628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:33.344245 31628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:41311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:33.345734 31628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:33.347149 31628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:33.361225 31633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:33.361294 31634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:33.361207 31636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:33.362720 31628 server_base.cc:1034] running on GCE node
I20260928 15:51:33.363520 31628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:33.364665 31628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:33.365909 31628 hybrid_clock.cc:648] HybridClock initialized: now 1790610693365840 us; error 64 us; skew 500 ppm
I20260928 15:51:33.366254 31628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:33 dist-test-slave-1vw0 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610693, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:33.369591 31628 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:51:33.371191 31628 webserver.cc:460] Webserver started at http://127.22.157.193:37351/ using document root <none> and password file <none>
I20260928 15:51:33.371869 31628 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:33.371970 31628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:33.372246 31628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:33.375259 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "3e798e9f3eea4191b390cc47969031ae"
format_stamp: "Formatted at 2026-09-28 15:51:33 on dist-test-slave-1vw0"
server_key: "73d67dc974894e50fffe63130cc0ac22"
server_key_iv: "0d187ed77a799fe8d461403038daafbf"
server_key_version: "encryptionkey@0"
I20260928 15:51:33.376000 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "3e798e9f3eea4191b390cc47969031ae"
format_stamp: "Formatted at 2026-09-28 15:51:33 on dist-test-slave-1vw0"
server_key: "73d67dc974894e50fffe63130cc0ac22"
server_key_iv: "0d187ed77a799fe8d461403038daafbf"
server_key_version: "encryptionkey@0"
I20260928 15:51:33.381312 31628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 15:51:33.384881 31643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:33.386281 31628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:33.386420 31628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "3e798e9f3eea4191b390cc47969031ae"
format_stamp: "Formatted at 2026-09-28 15:51:33 on dist-test-slave-1vw0"
server_key: "73d67dc974894e50fffe63130cc0ac22"
server_key_iv: "0d187ed77a799fe8d461403038daafbf"
server_key_version: "encryptionkey@0"
I20260928 15:51:33.386669 31628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:33.403160 31628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:33.407526 31628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:33.407837 31628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:33.408914 31628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:33.410320 31628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:33.410395 31628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:33.410560 31628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:33.410626 31628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:33.444624 31628 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:45907
I20260928 15:51:33.444675 31755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:45907 every 8 connection(s)
I20260928 15:51:33.446206 31628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:33.451524 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 31628
I20260928 15:51:33.451742 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:51:33.452579 23159 external_mini_cluster.cc:1428] Setting key 59fc57e35ea3647ad5d4493926ea8608
Sep 28 15:51:33 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610693, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:51:33.471956 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:33.448935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:52095 (local address 127.22.157.254:43797)
0928 15:51:33.449447 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:33.449451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:33.450602 (+ 1151us) server_negotiation.cc:407] Connection header received
0928 15:51:33.453357 (+ 2755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:33.453363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:33.453471 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:33.453594 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:33.455639 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:33.456364 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:33.459071 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:33.459473 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:33.459586 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:33.464304 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:33.464313 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:33.464320 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:33.464371 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:33.468142 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:33.469125 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:33.469134 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:33.469141 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:33.469320 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:33.469842 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:33.469849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:33.469855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:33.470263 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:33.470428 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:33.471181 (+ 753us) server_negotiation.cc:299] Negotiation successful
0928 15:51:33.471463 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
I20260928 15:51:33.472781 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:33.449628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:52095)
0928 15:51:33.450432 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:33.450476 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:33.453018 (+ 2542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:33.453924 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:33.453933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:33.454540 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:33.455435 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:33.455464 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:33.456545 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:33.456554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:33.458874 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:33.458905 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:33.459727 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:33.459737 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:33.460112 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:33.461011 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:33.461048 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:33.463998 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:33.468352 (+ 4354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:33.468362 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:33.468378 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:33.468964 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:33.469479 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:33.469483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:33.469485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:33.469714 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:33.470385 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:33.470392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:33.470757 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 15:51:33.471005 (+ 248us) client_negotiation.cc:240] Negotiation successful
0928 15:51:33.471432 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":281,"threads_started":1}
I20260928 15:51:33.474551 31756 heartbeater.cc:347] Connected to a master server at 127.22.157.254:43797
I20260928 15:51:33.474912 31756 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:33.475888 31756 heartbeater.cc:511] Master 127.22.157.254:43797 requested a full tablet report, sending...
I20260928 15:51:33.478873 31566 ts_manager.cc:194] Registered new tserver with Master: 3e798e9f3eea4191b390cc47969031ae (127.22.157.193:45907)
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
I20260928 15:51:33.481132 31566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:52095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:51:33.513337 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43797
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:41311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:33.860148 31763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:33.860479 31763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:33.860649 31763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:33.868251 31763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:33.868381 31763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:33.868582 31763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:51:33.878589 31763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:41311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:33.879949 31763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:33.881513 31763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:33.894838 31771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:33.894909 31768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:33.894901 31769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:33.895195 31763 server_base.cc:1034] running on GCE node
I20260928 15:51:33.896150 31763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:33.897153 31763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:33.898379 31763 hybrid_clock.cc:648] HybridClock initialized: now 1790610693898338 us; error 47 us; skew 500 ppm
I20260928 15:51:33.898715 31763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:33 dist-test-slave-1vw0 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610693, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:33.901898 31763 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:51:33.903345 31763 webserver.cc:460] Webserver started at http://127.22.157.194:42755/ using document root <none> and password file <none>
I20260928 15:51:33.904011 31763 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:33.904122 31763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:33.904373 31763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:33.907447 31763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "1a0d7f3f55074a219a2064e3ba966d21"
format_stamp: "Formatted at 2026-09-28 15:51:33 on dist-test-slave-1vw0"
server_key: "1412dc7fcd0d9ddfa8866c1bc46ec68a"
server_key_iv: "92197ae3fd22169e1343e7de62590c09"
server_key_version: "encryptionkey@0"
I20260928 15:51:33.908191 31763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "1a0d7f3f55074a219a2064e3ba966d21"
format_stamp: "Formatted at 2026-09-28 15:51:33 on dist-test-slave-1vw0"
server_key: "1412dc7fcd0d9ddfa8866c1bc46ec68a"
server_key_iv: "92197ae3fd22169e1343e7de62590c09"
server_key_version: "encryptionkey@0"
I20260928 15:51:33.913357 31763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:51:33.916706 31778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:33.918102 31763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:33.918286 31763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "1a0d7f3f55074a219a2064e3ba966d21"
format_stamp: "Formatted at 2026-09-28 15:51:33 on dist-test-slave-1vw0"
server_key: "1412dc7fcd0d9ddfa8866c1bc46ec68a"
server_key_iv: "92197ae3fd22169e1343e7de62590c09"
server_key_version: "encryptionkey@0"
I20260928 15:51:33.918495 31763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:33.936410 31763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:33.940963 31763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:33.941264 31763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:33.942234 31763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:33.943645 31763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:33.943745 31763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:33.943898 31763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:33.943970 31763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:33.978474 31763 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:44923
I20260928 15:51:33.978519 31890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:44923 every 8 connection(s)
I20260928 15:51:33.979770 31763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:51:33.987982 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 31763
I20260928 15:51:33.988157 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:51:33.988813 23159 external_mini_cluster.cc:1428] Setting key 3e38f655e727b7f582ac4631ee44eca0
Sep 28 15:51:33 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610693, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:51:34.000885 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:33.982276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:48925 (local address 127.22.157.254:43797)
0928 15:51:33.982699 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:33.982703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:33.983746 (+ 1043us) server_negotiation.cc:407] Connection header received
0928 15:51:33.985648 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:33.985654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:33.985747 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:33.985868 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:33.987385 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:33.988151 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:33.990425 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:33.991156 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:33.991295 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:33.995030 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:33.995035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:33.995038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:33.995070 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:33.997734 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:33.998490 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:33.998500 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:33.998503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:33.998629 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:33.999054 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:33.999059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:33.999062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:33.999338 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:33.999491 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:34.000230 (+ 739us) server_negotiation.cc:299] Negotiation successful
0928 15:51:34.000476 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":320,"thread_start_us":193,"threads_started":1}
I20260928 15:51:34.001940 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:33.982860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:48925)
0928 15:51:33.983582 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:33.983620 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:33.985373 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:33.986112 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:33.986121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:33.986643 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:33.987216 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:33.987233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:33.988356 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:33.988361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:33.990250 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:33.990268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:33.991385 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:33.991393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:33.991710 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:33.992393 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:33.992415 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:33.994797 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:33.997888 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:33.997897 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:33.997914 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:33.998353 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:33.998737 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:33.998742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:33.998746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:33.998934 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:33.999462 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:33.999470 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:33.999846 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 15:51:34.000119 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 15:51:34.000702 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":277,"threads_started":1}
I20260928 15:51:34.003415 31891 heartbeater.cc:347] Connected to a master server at 127.22.157.254:43797
I20260928 15:51:34.003758 31891 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:34.004702 31891 heartbeater.cc:511] Master 127.22.157.254:43797 requested a full tablet report, sending...
I20260928 15:51:34.006338 31566 ts_manager.cc:194] Registered new tserver with Master: 1a0d7f3f55074a219a2064e3ba966d21 (127.22.157.194:44923)
I20260928 15:51:34.007171 31566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:48925
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:51:34.044106 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43797
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:41311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:34.374611 31897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:34.374984 31897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:34.375165 31897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:34.382594 31897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:34.382788 31897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:34.382985 31897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:51:34.393360 31897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:41311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:34.394796 31897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:34.396175 31897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:34.409812 31902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:34.409844 31903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:34.409915 31906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:34.410775 31897 server_base.cc:1034] running on GCE node
I20260928 15:51:34.411441 31897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:34.412454 31897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:34.413733 31897 hybrid_clock.cc:648] HybridClock initialized: now 1790610694413649 us; error 69 us; skew 500 ppm
I20260928 15:51:34.414143 31897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:34 dist-test-slave-1vw0 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610694, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:34.417683 31897 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:51:34.419179 31897 webserver.cc:460] Webserver started at http://127.22.157.195:38035/ using document root <none> and password file <none>
I20260928 15:51:34.419924 31897 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:34.420071 31897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:34.420385 31897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:34.423554 31897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "d859bae3c38348699afeed2e1424b0d7"
format_stamp: "Formatted at 2026-09-28 15:51:34 on dist-test-slave-1vw0"
server_key: "a62042f49463a46fab9209832666c01e"
server_key_iv: "d32392dad12f257dfd963e25e3722642"
server_key_version: "encryptionkey@0"
I20260928 15:51:34.424350 31897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "d859bae3c38348699afeed2e1424b0d7"
format_stamp: "Formatted at 2026-09-28 15:51:34 on dist-test-slave-1vw0"
server_key: "a62042f49463a46fab9209832666c01e"
server_key_iv: "d32392dad12f257dfd963e25e3722642"
server_key_version: "encryptionkey@0"
I20260928 15:51:34.429960 31897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:51:34.433636 31913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:34.435159 31897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:34.435362 31897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "d859bae3c38348699afeed2e1424b0d7"
format_stamp: "Formatted at 2026-09-28 15:51:34 on dist-test-slave-1vw0"
server_key: "a62042f49463a46fab9209832666c01e"
server_key_iv: "d32392dad12f257dfd963e25e3722642"
server_key_version: "encryptionkey@0"
I20260928 15:51:34.435607 31897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:34.458705 31897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:34.463939 31897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:34.464263 31897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:34.465400 31897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:34.467286 31897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:34.467420 31897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:34.467556 31897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:34.467628 31897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:34.484968 31756 heartbeater.cc:503] Master 127.22.157.254:43797 was elected leader, sending a full tablet report...
I20260928 15:51:34.503436 31897 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:36459
I20260928 15:51:34.503665 32025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:36459 every 8 connection(s)
I20260928 15:51:34.504819 31897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:51:34.508170 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 31897
I20260928 15:51:34.508363 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:51:34.509014 23159 external_mini_cluster.cc:1428] Setting key 8c0a68debe498e4581b823a90c4cea34
Sep 28 15:51:34 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610694, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:51:34.526253 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:34.507468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:37575 (local address 127.22.157.254:43797)
0928 15:51:34.507965 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:34.507969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:34.509566 (+ 1597us) server_negotiation.cc:407] Connection header received
0928 15:51:34.511191 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:34.511197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:34.511272 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:34.511365 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:34.513270 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.514181 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.515738 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.516209 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.516321 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:34.519868 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:34.519873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:34.519876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:34.519928 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:34.522975 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:34.523893 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:34.523899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:34.523903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:34.524052 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:34.524591 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:34.524598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:34.524604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:34.524977 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:34.525098 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:34.525803 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 15:51:34.525977 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":260,"threads_started":1}
I20260928 15:51:34.527251 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:34.508076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:37575)
0928 15:51:34.508961 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:34.509004 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:34.510919 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:34.511635 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:34.511646 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:34.512414 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:34.513074 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:34.513092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.514336 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.514340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:34.515592 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:34.515605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.516432 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.516437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:34.516652 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:34.517422 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:34.517446 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:34.519651 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:34.523156 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:34.523163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:34.523177 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:34.523769 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:34.524177 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:34.524180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:34.524183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:34.524445 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:34.525097 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:34.525103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:34.525446 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 15:51:34.525687 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 15:51:34.526094 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":357,"threads_started":1}
I20260928 15:51:34.528947 32026 heartbeater.cc:347] Connected to a master server at 127.22.157.254:43797
I20260928 15:51:34.529295 32026 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:34.530130 32026 heartbeater.cc:511] Master 127.22.157.254:43797 requested a full tablet report, sending...
I20260928 15:51:34.531702 31566 ts_manager.cc:194] Registered new tserver with Master: d859bae3c38348699afeed2e1424b0d7 (127.22.157.195:36459)
I20260928 15:51:34.532581 31566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:37575
I20260928 15:51:34.537351 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab.
Entry for principal oryx/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab.
WARNING: no policy specified for oryx/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab.
Entry for principal oryx/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab.
WARNING: no policy specified for oryx/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab.
Entry for principal oryx/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab.
I20260928 15:51:34.695230 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 31628
Sep 28 15:51:34 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:51:34.712189 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:34.696542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.22.157.193:45907)
0928 15:51:34.697352 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:34.697360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:34.697387 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:51:34.697449 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:34.697457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:34.697680 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:34.697807 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:34.699202 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.700235 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.701505 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.702085 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.702181 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:34.706222 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:34.706234 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:34.706249 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:34.706289 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:34.709305 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:34.710039 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:34.710046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:34.710049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:34.710194 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:34.710579 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:34.710584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:34.710586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:34.710946 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:34.711280 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:34.711533 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:51:34.711821 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":459,"threads_started":1}
I20260928 15:51:34.713598 31751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:34.796960 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 31628
I20260928 15:51:34.814749 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 31763
Sep 28 15:51:34 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:51:34.830757 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:34.815837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.22.157.194:44923)
0928 15:51:34.816371 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:34.816378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:34.816404 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:51:34.816469 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:34.816475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:34.816716 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:34.816915 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:34.818106 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.819141 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.820438 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.821093 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.821192 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:34.825569 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:34.825582 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:34.825596 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:34.825633 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:34.828311 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:34.829013 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:34.829020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:34.829022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:34.829167 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:34.829466 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:34.829469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:34.829471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:34.829799 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:34.830015 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:34.830260 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 15:51:34.830455 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":308,"threads_started":1}
I20260928 15:51:34.832080 31886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:34.915246 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 31763
I20260928 15:51:34.932077 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 31897
Sep 28 15:51:34 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:51:34.949793 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:34.933254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.22.157.195:36459)
0928 15:51:34.933805 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:34.933812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:34.933845 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:51:34.933904 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:34.933911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:34.934144 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:34.934273 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:34.935612 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.936693 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.938221 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:34.938779 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:34.938880 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:34.943676 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:34.943688 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:34.943703 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:34.943745 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:34.946808 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:34.947522 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:34.947530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:34.947533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:34.947691 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:34.948089 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:34.948097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:34.948100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:34.948466 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:34.948812 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:34.949145 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 15:51:34.949428 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":264,"threads_started":1}
I20260928 15:51:34.951220 32021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:35.036585 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 31897
I20260928 15:51:35.052980 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 31533
I20260928 15:51:35.054376 31596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:35.160233 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 31533
I20260928 15:51:35.181659 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43797
--webserver_interface=127.22.157.254
--webserver_port=36909
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:41311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:43797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:35.522405 32045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:35.522763 32045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:35.522927 32045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:35.531247 32045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:35.531383 32045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:35.531477 32045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:35.531561 32045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:35.531639 32045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:35.542224 32045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:41311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:43797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=36909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:35.543745 32045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:35.545185 32045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:35.558218 32051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:35.558394 32050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:35.558374 32053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:35.559751 32045 server_base.cc:1034] running on GCE node
I20260928 15:51:35.560633 32045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:35.562239 32045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:35.563536 32045 hybrid_clock.cc:648] HybridClock initialized: now 1790610695563464 us; error 58 us; skew 500 ppm
I20260928 15:51:35.563946 32045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:35 dist-test-slave-1vw0 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610695, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:35.567364 32045 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:51:35.569315 32045 webserver.cc:460] Webserver started at http://127.22.157.254:36909/ using document root <none> and password file <none>
I20260928 15:51:35.570079 32045 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:35.570204 32045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:35.577334 32045 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:51:35.581075 32060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:35.583274 32045 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:51:35.583467 32045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "5e41847d15dd4410b4ffc33b153e232f"
format_stamp: "Formatted at 2026-09-28 15:51:32 on dist-test-slave-1vw0"
server_key: "529e3b6f4cbff12822962cfda74b0713"
server_key_iv: "4db362ecb33f247dbe3e92e26b9cb031"
server_key_version: "encryptionkey@0"
I20260928 15:51:35.584551 32045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:35.604394 32045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:35.609985 32045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:35.610260 32045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:35.631846 32045 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:43797
I20260928 15:51:35.631868 32111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:43797 every 8 connection(s)
I20260928 15:51:35.633690 32045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:35.636904 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 32045
I20260928 15:51:35.638011 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:45907
--local_ip_for_outbound_sockets=127.22.157.193
--tserver_master_addrs=127.22.157.254:43797
--webserver_port=37351
--webserver_interface=127.22.157.193
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:41311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.157.193
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:51:35.643930 32112 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:51:35.649178 32112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: Bootstrap starting.
I20260928 15:51:35.670594 32112 log.cc:824] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:35.680539 32112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:51:35.681180 32112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: Bootstrap complete.
I20260928 15:51:35.691068 32112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } }
I20260928 15:51:35.691972 32112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e41847d15dd4410b4ffc33b153e232f, State: Initialized, Role: FOLLOWER
I20260928 15:51:35.692680 32112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } }
I20260928 15:51:35.692991 32112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:35.693114 32112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:35.693316 32112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:51:35.697391 32112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } }
I20260928 15:51:35.697959 32112 leader_election.cc:302] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e41847d15dd4410b4ffc33b153e232f; no voters:
I20260928 15:51:35.698642 32112 leader_election.cc:288] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:51:35.698794 32116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:51:35.700151 32116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [term 2 LEADER]: Becoming Leader. State: Replica: 5e41847d15dd4410b4ffc33b153e232f, State: Running, Role: LEADER
I20260928 15:51:35.700865 32116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } }
I20260928 15:51:35.701198 32112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:35.703512 32118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e41847d15dd4410b4ffc33b153e232f. Latest consensus state: current_term: 2 leader_uuid: "5e41847d15dd4410b4ffc33b153e232f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } } }
I20260928 15:51:35.703876 32118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:35.704429 32117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e41847d15dd4410b4ffc33b153e232f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e41847d15dd4410b4ffc33b153e232f" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43797 } } }
I20260928 15:51:35.704772 32117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:35.716816 32123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:35.723171 32123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:35.725158 32123 catalog_manager.cc:1259] Loaded cluster ID: 5d790752e60f4c25b8c5ca356645534d
I20260928 15:51:35.725401 32123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:35.730923 32123 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:35.733181 32123 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e41847d15dd4410b4ffc33b153e232f: Loaded TSK: 0
I20260928 15:51:35.734522 32123 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:51:36.026243 32114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:36.026604 32114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:36.026769 32114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:36.034363 32114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:36.034535 32114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:36.034747 32114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:36.045212 32114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:41311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.193.keytab
--principal=oryx/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:45907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=37351
--tserver_master_addrs=127.22.157.254:43797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:36.046949 32114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:36.048305 32114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:36.063257 32139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:36.063324 32138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:36.063293 32141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:36.065124 32114 server_base.cc:1034] running on GCE node
I20260928 15:51:36.065881 32114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:36.066993 32114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:36.068225 32114 hybrid_clock.cc:648] HybridClock initialized: now 1790610696068158 us; error 66 us; skew 500 ppm
I20260928 15:51:36.068565 32114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:36.072007 32114 init.cc:377] Logged in from keytab as oryx/127.22.157.193@KRBTEST.COM (short username oryx)
I20260928 15:51:36.073735 32114 webserver.cc:460] Webserver started at http://127.22.157.193:37351/ using document root <none> and password file <none>
I20260928 15:51:36.074498 32114 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:36.074666 32114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:36.080601 32114 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 15:51:36.083621 32148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:36.085242 32114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:36.085427 32114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "3e798e9f3eea4191b390cc47969031ae"
format_stamp: "Formatted at 2026-09-28 15:51:33 on dist-test-slave-1vw0"
server_key: "73d67dc974894e50fffe63130cc0ac22"
server_key_iv: "0d187ed77a799fe8d461403038daafbf"
server_key_version: "encryptionkey@0"
I20260928 15:51:36.086474 32114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:36.113168 32114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:36.117555 32114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:36.117879 32114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:36.118916 32114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:36.120398 32114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:36.120522 32114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:36.120654 32114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:36.120762 32114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:36.154961 32114 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:45907
I20260928 15:51:36.155057 32262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:45907 every 8 connection(s)
I20260928 15:51:36.156589 32114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:36.157825 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 32114
I20260928 15:51:36.158685 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:44923
--local_ip_for_outbound_sockets=127.22.157.194
--tserver_master_addrs=127.22.157.254:43797
--webserver_port=42755
--webserver_interface=127.22.157.194
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:41311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.157.194
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:36.185640 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:36.160551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:37371)
0928 15:51:36.163666 (+ 3115us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:36.163718 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:36.166525 (+ 2807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:36.168935 (+ 2410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:36.168970 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:36.169877 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:36.170811 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.170860 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.173445 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.173451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.175690 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.175710 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.177739 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.177750 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.178119 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.179121 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:36.179154 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:36.184210 (+ 5056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2781,"thread_start_us":2576,"threads_started":1}
W20260928 15:51:36.185808 32265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:36.160554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:37371 (local address 127.22.157.254:43797)
0928 15:51:36.162541 (+ 1987us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:36.162566 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:36.163867 (+ 1301us) server_negotiation.cc:407] Connection header received
0928 15:51:36.167231 (+ 3364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:36.167255 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:36.167732 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:36.168086 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:36.171177 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.173254 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.175946 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.177265 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.177482 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.184620 (+ 7138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:37371: BlockingRecv error: recv got EOF from 127.22.157.193:37371 (error 108)
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":288,"threads_started":1}
W20260928 15:51:36.186443 32263 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:43797 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.157.254:43797: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:36.199538 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:36.187492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:58319)
0928 15:51:36.188184 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:36.188208 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:36.188373 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:36.188850 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:36.188871 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:36.189381 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:36.189622 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.189633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.191342 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.191347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.192974 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.192993 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.193995 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.194014 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.194266 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.195064 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:36.195095 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:36.199192 (+ 4097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":288,"threads_started":1}
W20260928 15:51:36.199817 32265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:36.187654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:58319 (local address 127.22.157.254:43797)
0928 15:51:36.187867 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:36.187872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:36.188306 (+ 434us) server_negotiation.cc:407] Connection header received
0928 15:51:36.188521 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:36.188526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:36.188636 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:36.188797 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:36.189770 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.191159 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.193169 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.193804 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.193957 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.199516 (+ 5559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:58319: BlockingRecv error: recv got EOF from 127.22.157.193:58319 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:36.216151 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:36.200514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:58807)
0928 15:51:36.200775 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:36.200798 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:36.200927 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:36.205860 (+ 4933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:36.205866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:36.206313 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:36.206524 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.206538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.208236 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.208243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.209807 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.209828 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.211077 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.211085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.211274 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.212018 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:36.212041 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:36.215823 (+ 3782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 15:51:36.216928 32263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:51:36.216998 32265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:36.205203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:58807 (local address 127.22.157.254:43797)
0928 15:51:36.205417 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:36.205425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:36.205452 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:51:36.205531 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:36.205538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:36.205650 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:36.205786 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:36.206714 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.208045 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.210043 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.210669 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.210800 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.216683 (+ 5883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:58807: BlockingRecv error: recv got EOF from 127.22.157.193:58807 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 15:51:36.519809 32267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:36.520146 32267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:36.520310 32267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:36.527720 32267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:36.527889 32267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:36.528064 32267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:51:36.538208 32267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:41311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.194.keytab
--principal=oryx/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:44923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=42755
--tserver_master_addrs=127.22.157.254:43797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:36.539660 32267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:36.541149 32267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:36.558864 32273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:36.559008 32276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:36.559319 32274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:36.559957 32267 server_base.cc:1034] running on GCE node
I20260928 15:51:36.560881 32267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:36.562292 32267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:36.563555 32267 hybrid_clock.cc:648] HybridClock initialized: now 1790610696563485 us; error 53 us; skew 500 ppm
I20260928 15:51:36.564043 32267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610696, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:36.568819 32267 init.cc:377] Logged in from keytab as oryx/127.22.157.194@KRBTEST.COM (short username oryx)
I20260928 15:51:36.571041 32267 webserver.cc:460] Webserver started at http://127.22.157.194:42755/ using document root <none> and password file <none>
I20260928 15:51:36.572080 32267 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:36.572250 32267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:36.581413 32267 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:51:36.586010 32283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:36.588230 32267 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:51:36.588455 32267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "1a0d7f3f55074a219a2064e3ba966d21"
format_stamp: "Formatted at 2026-09-28 15:51:33 on dist-test-slave-1vw0"
server_key: "1412dc7fcd0d9ddfa8866c1bc46ec68a"
server_key_iv: "92197ae3fd22169e1343e7de62590c09"
server_key_version: "encryptionkey@0"
I20260928 15:51:36.590369 32267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:36.618211 32267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:36.625427 32267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:36.625797 32267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:36.627311 32267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:36.629657 32267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:36.629810 32267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:36.629951 32267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:36.630057 32267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:36.672291 32267 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:44923
I20260928 15:51:36.672302 32396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:44923 every 8 connection(s)
I20260928 15:51:36.674896 32267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:51:36.675776 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 32267
I20260928 15:51:36.676544 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:36459
--local_ip_for_outbound_sockets=127.22.157.195
--tserver_master_addrs=127.22.157.254:43797
--webserver_port=38035
--webserver_interface=127.22.157.195
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:41311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.157.195
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:51:36.709052 32265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:36.679674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:46573 (local address 127.22.157.254:43797)
0928 15:51:36.679883 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:36.679892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:36.689501 (+ 9609us) server_negotiation.cc:407] Connection header received
0928 15:51:36.693389 (+ 3888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:36.693399 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:36.693531 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:36.693663 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:36.696743 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.698087 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.700837 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.701717 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.701897 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.708684 (+ 6787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:46573: BlockingRecv error: recv got EOF from 127.22.157.194:46573 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:51:36.709721 32400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:36.679013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:46573)
0928 15:51:36.689218 (+ 10205us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:36.689301 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:36.692835 (+ 3534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:36.694229 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:36.694246 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:36.695291 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:36.696431 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.696469 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.698319 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.698328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.700550 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.700569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.702114 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.702124 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.702448 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.703350 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:36.703384 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:36.708365 (+ 4981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":9771,"thread_start_us":14194,"threads_started":1}
W20260928 15:51:36.710711 32397 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:43797 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.157.254:43797: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:36.721201 32400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:36.711428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:33499)
0928 15:51:36.711705 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:36.711724 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:36.711833 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:36.712215 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:36.712219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:36.712666 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:36.712902 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.712914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.714525 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.714530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.715590 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.715604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.716715 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.716719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.716965 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.717705 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:36.717723 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:36.720963 (+ 3240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153}
W20260928 15:51:36.721954 32265 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:36.711604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:33499 (local address 127.22.157.254:43797)
0928 15:51:36.711765 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:36.711771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:36.711791 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:51:36.711970 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:36.711975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:36.712054 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:36.712162 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:36.713060 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.714358 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.715746 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.716556 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.716685 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.721302 (+ 4617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:33499: BlockingRecv error: recv got EOF from 127.22.157.194:33499 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:36 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:36.730762 32400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:36.722135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:57399)
0928 15:51:36.722295 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:36.722321 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:36.722463 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:36.722961 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:36.722967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:36.723305 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:36.723456 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.723464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.724520 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.724526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.725549 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:36.725562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.726281 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.726287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:36.726502 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.727131 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:36.727147 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:36.730451 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:51:36.731093 32397 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:51:36.731225 32402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:36.722190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:57399 (local address 127.22.157.254:43797)
0928 15:51:36.722680 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:36.722686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:36.722709 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:51:36.722760 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:36.722763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:36.722818 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:36.722932 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:36.723611 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.724378 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.725702 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:36.726130 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:36.726264 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:36.730715 (+ 4451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:57399: BlockingRecv error: recv got EOF from 127.22.157.194:57399 (error 108)
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":257,"threads_started":1}
W20260928 15:51:37.036852 32401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:37.037168 32401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:37.037355 32401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:37.045034 32401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:37.045169 32401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:37.045352 32401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:51:37.055657 32401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:41311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/krb5kdc/oryx_127.22.157.195.keytab
--principal=oryx/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:36459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=38035
--tserver_master_addrs=127.22.157.254:43797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:37.057183 32401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:37.058701 32401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:37.073244 32408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:37.073372 32407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:37.073338 32410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:37.074182 32401 server_base.cc:1034] running on GCE node
I20260928 15:51:37.074994 32401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:37.076077 32401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:37.077327 32401 hybrid_clock.cc:648] HybridClock initialized: now 1790610697077286 us; error 41 us; skew 500 ppm
I20260928 15:51:37.077723 32401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610697, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:37.081252 32401 init.cc:377] Logged in from keytab as oryx/127.22.157.195@KRBTEST.COM (short username oryx)
I20260928 15:51:37.082795 32401 webserver.cc:460] Webserver started at http://127.22.157.195:38035/ using document root <none> and password file <none>
I20260928 15:51:37.083545 32401 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:37.083670 32401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:37.090344 32401 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:51:37.093281 32417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:37.094801 32401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:51:37.094988 32401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "d859bae3c38348699afeed2e1424b0d7"
format_stamp: "Formatted at 2026-09-28 15:51:34 on dist-test-slave-1vw0"
server_key: "a62042f49463a46fab9209832666c01e"
server_key_iv: "d32392dad12f257dfd963e25e3722642"
server_key_version: "encryptionkey@0"
I20260928 15:51:37.096061 32401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:37.131312 32401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:37.135890 32401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:37.136183 32401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:37.137259 32401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:37.138653 32401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:37.138741 32401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:37.138911 32401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:37.139000 32401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:37.174363 32401 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:36459
I20260928 15:51:37.174443 32529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:36459 every 8 connection(s)
I20260928 15:51:37.175966 32401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:51:37.180972 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 32401
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 15:51:37.194415 32402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:37.178642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:47585 (local address 127.22.157.254:43797)
0928 15:51:37.178834 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:37.178839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:37.180068 (+ 1229us) server_negotiation.cc:407] Connection header received
0928 15:51:37.182482 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:37.182489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:37.182591 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:37.182714 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:37.184620 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.185635 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.187540 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.188163 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.188301 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.194127 (+ 5826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:47585: BlockingRecv error: recv got EOF from 127.22.157.195:47585 (error 108)
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":68}
I20260928 15:51:37.195012 32532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:37.179135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:47585)
0928 15:51:37.179886 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:37.179931 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:37.182080 (+ 2149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:37.183108 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:37.183124 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:37.183787 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:37.184414 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.184440 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.185839 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.185843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.187382 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.187399 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.188438 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.188446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.188850 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.189592 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:37.189614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:37.193848 (+ 4234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":289,"threads_started":1}
W20260928 15:51:37.195752 32530 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:43797 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.157.254:43797: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
I20260928 15:51:37.199136 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:37.182025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.22.157.254:43797)
0928 15:51:37.182213 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:37.182217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:37.182804 (+ 587us) server_negotiation.cc:407] Connection header received
0928 15:51:37.182963 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:37.182968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:37.183046 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:37.183155 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:37.184697 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.185628 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.186889 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.187605 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.187771 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.191083 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:37.191092 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:37.191110 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:37.191149 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:37.194989 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:37.195976 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:37.195985 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:37.196002 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:37.196237 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:37.196832 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:37.196840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:37.196845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:37.197576 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:37.197915 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:37.198370 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 15:51:37.198735 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":85}
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:37.205896 32532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:37.196450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:37323)
0928 15:51:37.196773 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:37.196795 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:37.196913 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:37.197247 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:37.197254 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:37.197687 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:37.197891 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.197901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.199239 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.199247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.200419 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.200436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.201296 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.201302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.201481 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.202601 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:37.202629 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:37.205676 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260928 15:51:37.206234 32402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:37.196557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:37323 (local address 127.22.157.254:43797)
0928 15:51:37.196792 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:37.196798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:37.196853 (+ 55us) server_negotiation.cc:407] Connection header received
0928 15:51:37.197032 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:37.197035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:37.197110 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:37.197184 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:37.198049 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.199060 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.200553 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.201154 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.201252 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.205906 (+ 4654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:37323: BlockingRecv error: recv got EOF from 127.22.157.195:37323 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":61}
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:37.215008 32532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:37.206735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:48133)
0928 15:51:37.206913 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:37.206939 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:37.207026 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:37.207445 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:37.207448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:37.207759 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:37.207919 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.207927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.209026 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.209031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.210018 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.210031 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.210747 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.210751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.210878 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.211450 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:37.211467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:37.214667 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 15:51:37.215152 32402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:37.206879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:48133 (local address 127.22.157.254:43797)
0928 15:51:37.207161 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:37.207168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:37.207184 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:51:37.207234 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:37.207237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:37.207319 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:37.207404 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:37.208075 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.208901 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.210161 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.210616 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.210719 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.214911 (+ 4192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:48133: BlockingRecv error: recv got EOF from 127.22.157.195:48133 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
W20260928 15:51:37.215447 32530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:37.226092 32534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:37.217448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:33055)
0928 15:51:37.218006 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:37.218023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:37.218174 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:37.218468 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:37.218471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:37.218903 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:37.219120 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.219130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.220216 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.220220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.221259 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.221273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.221967 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.221971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.222132 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.222726 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:37.222744 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:37.225813 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":330,"threads_started":1}
W20260928 15:51:37.226320 32402 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:37.217531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:33055 (local address 127.22.157.254:43797)
0928 15:51:37.217699 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:37.217703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:37.218093 (+ 390us) server_negotiation.cc:407] Connection header received
0928 15:51:37.218277 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:37.218280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:37.218342 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:37.218423 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:37.219250 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.220073 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.221388 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.221803 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.221901 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.226071 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:33055: BlockingRecv error: recv got EOF from 127.22.157.193:33055 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:37 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:37.749462 32535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:37.731918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:38113)
0928 15:51:37.732498 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:37.732516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:37.732647 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:37.733158 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:37.733163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:37.733759 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:37.734055 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.734068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.735475 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.735480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.736581 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:37.736598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.737359 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.737364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:37.737542 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.745165 (+ 7623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:37.745190 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:37.749212 (+ 4022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":306,"threads_started":1}
W20260928 15:51:37.749912 32536 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:37.731963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:38113 (local address 127.22.157.254:43797)
0928 15:51:37.732441 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:37.732446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:37.732618 (+ 172us) server_negotiation.cc:407] Connection header received
0928 15:51:37.732799 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:37.732805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:37.732899 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:37.733028 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:37.734238 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.735300 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.736768 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:37.737163 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:37.737262 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:37.749625 (+ 12363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:38113: BlockingRecv error: recv got EOF from 127.22.157.194:38113 (error 108)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":268,"threads_started":1}
Sep 28 15:51:38 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:38 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:38.225474 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:38.216273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:51869)
0928 15:51:38.216852 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:38.216873 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:38.217024 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:38.217427 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:38.217434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:38.218036 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:38.218254 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:38.218264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.219309 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.219313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:38.220271 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:38.220283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.221126 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.221131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:38.221267 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:38.222002 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:38.222030 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:38.225227 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":263,"threads_started":1}
W20260928 15:51:38.225769 32536 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:38.216592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:51869 (local address 127.22.157.254:43797)
0928 15:51:38.216815 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:38.216819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:38.216963 (+ 144us) server_negotiation.cc:407] Connection header received
0928 15:51:38.217163 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:38.217170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:38.217256 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:38.217366 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:38.218393 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.219187 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.220413 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.220933 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.221033 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:38.225456 (+ 4423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:51869: BlockingRecv error: recv got EOF from 127.22.157.195:51869 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 15:51:38 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:38 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:38.235913 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:38.227030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:36893)
0928 15:51:38.227542 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:38.227559 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:38.227672 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:38.228072 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:38.228076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:38.228609 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:38.228860 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:38.228876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.230260 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.230267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:38.231209 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:38.231222 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.232035 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.232039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:38.232156 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:38.232807 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:38.232831 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:38.235675 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":280,"threads_started":1}
W20260928 15:51:38.236141 32536 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:38.227212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:36893 (local address 127.22.157.254:43797)
0928 15:51:38.227386 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:38.227390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:38.227653 (+ 263us) server_negotiation.cc:407] Connection header received
0928 15:51:38.227806 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:38.227809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:38.227883 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:38.228008 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:38.229060 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.230073 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.231392 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.231865 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.231975 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:38.235882 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:36893: BlockingRecv error: recv got EOF from 127.22.157.193:36893 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 15:51:38 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:38 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:38.759665 32540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:38.750570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:54739)
0928 15:51:38.751193 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:38.751211 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:38.751329 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:38.751710 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:38.751714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:38.752127 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:38.752302 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:38.752310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.753483 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.753488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:38.754511 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:38.754527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.755403 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.755408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:38.755529 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:38.756237 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:38.756256 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:38.759281 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":274,"threads_started":1}
W20260928 15:51:38.759765 32541 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:38.750734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:54739 (local address 127.22.157.254:43797)
0928 15:51:38.751287 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:38.751292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:38.751313 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:51:38.751454 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:38.751458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:38.751528 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:38.751669 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:38.752477 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.753301 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.754693 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:38.755235 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:38.755362 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:38.759514 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:54739: BlockingRecv error: recv got EOF from 127.22.157.194:54739 (error 108)
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":297,"threads_started":1}
Sep 28 15:51:39 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:39 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:39.235615 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:39.226568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:55739)
0928 15:51:39.227147 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:39.227164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:39.227279 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:39.227643 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:39.227645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:39.228061 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:39.228231 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:39.228240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.229329 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.229333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:39.230275 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:39.230287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.231160 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.231164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:39.231336 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:39.232122 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:39.232150 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:39.235246 (+ 3096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":327,"threads_started":1}
W20260928 15:51:39.235713 32541 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:39.226785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:55739 (local address 127.22.157.254:43797)
0928 15:51:39.226981 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:39.226986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:39.227241 (+ 255us) server_negotiation.cc:407] Connection header received
0928 15:51:39.227429 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:39.227433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:39.227506 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:39.227597 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:39.228389 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.229200 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.230402 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.231007 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.231133 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:39.235463 (+ 4330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:55739: BlockingRecv error: recv got EOF from 127.22.157.195:55739 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 15:51:39 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:39 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:39.245121 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:39.236935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:34127)
0928 15:51:39.237461 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:39.237477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:39.237587 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:39.237874 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:39.237877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:39.238208 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:39.238374 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:39.238383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.239341 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.239345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:39.240302 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:39.240313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.241114 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.241119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:39.241241 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:39.241803 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:39.241820 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:39.244706 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":289,"threads_started":1}
W20260928 15:51:39.245203 32541 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:39.236984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:34127 (local address 127.22.157.254:43797)
0928 15:51:39.237149 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:39.237153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:39.237547 (+ 394us) server_negotiation.cc:407] Connection header received
0928 15:51:39.237691 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:39.237695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:39.237751 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:39.237830 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:39.238500 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.239218 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.240427 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.240951 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.241047 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:39.244938 (+ 3891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:34127: BlockingRecv error: recv got EOF from 127.22.157.193:34127 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 15:51:39 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:39 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:39.769559 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:39.760901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:54801)
0928 15:51:39.761403 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:39.761421 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:39.761524 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:39.761977 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:39.761980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:39.762317 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:39.762485 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:39.762493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.763479 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.763482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:39.764357 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:39.764368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.765142 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.765147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:39.765289 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:39.765888 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:39.765906 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:39.769148 (+ 3242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":261,"threads_started":1}
W20260928 15:51:39.769649 32545 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:39.760982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:54801 (local address 127.22.157.254:43797)
0928 15:51:39.761717 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:39.761721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:39.761737 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:51:39.761782 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:39.761786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:39.761852 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:39.761939 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:39.762625 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.763372 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.764474 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:39.764985 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:39.765076 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:39.769363 (+ 4287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:54801: BlockingRecv error: recv got EOF from 127.22.157.194:54801 (error 108)
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":519,"threads_started":1}
Sep 28 15:51:40 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:40 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:40.246156 32546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:40.236830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:40411)
0928 15:51:40.237393 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:40.237410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:40.237584 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:40.237979 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:40.237983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:40.238563 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:40.238746 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:40.238757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.240331 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.240335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:40.241272 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:40.241284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.241989 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.241993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:40.242127 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:40.242805 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:40.242823 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:40.245747 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":293,"threads_started":1}
W20260928 15:51:40.246706 32545 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:40.236967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:40411 (local address 127.22.157.254:43797)
0928 15:51:40.237166 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:40.237170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:40.237509 (+ 339us) server_negotiation.cc:407] Connection header received
0928 15:51:40.237722 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:40.237729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:40.237800 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:40.237921 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:40.239036 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.240179 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.241420 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.241852 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.241959 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:40.246031 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:40411: BlockingRecv error: recv got EOF from 127.22.157.195:40411 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 15:51:40 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:40 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:40.255510 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:40.246256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:43043)
0928 15:51:40.247001 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:40.247023 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:40.247169 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:40.247485 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:40.247488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:40.247871 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:40.248029 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:40.248038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.249245 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.249249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:40.250299 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:40.250314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.251244 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.251249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:40.251427 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:40.252209 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:40.252228 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:40.255269 (+ 3041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":497,"threads_started":1}
W20260928 15:51:40.255885 32548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:40.246298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:43043 (local address 127.22.157.254:43797)
0928 15:51:40.246891 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:40.246897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:40.247121 (+ 224us) server_negotiation.cc:407] Connection header received
0928 15:51:40.247280 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:40.247284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:40.247352 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:40.247441 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:40.248200 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.249124 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.250475 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.251091 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.251187 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:40.255551 (+ 4364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:43043: BlockingRecv error: recv got EOF from 127.22.157.193:43043 (error 108)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":619,"threads_started":1}
Sep 28 15:51:40 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:40 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:40.779240 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:40.770593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:36521)
0928 15:51:40.771070 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:40.771089 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:40.771216 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:40.771651 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:40.771654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:40.772006 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:40.772157 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:40.772164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.773430 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.773434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:40.774482 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:40.774496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.775183 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.775187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:40.775308 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:40.775910 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:40.775933 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:40.778879 (+ 2946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
W20260928 15:51:40.779491 32550 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:40.770641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:36521 (local address 127.22.157.254:43797)
0928 15:51:40.771393 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:40.771398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:40.771413 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:51:40.771459 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:40.771462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:40.771520 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:40.771617 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:40.772453 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.773310 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.774636 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:40.775056 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:40.775159 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:40.779128 (+ 3969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:36521: BlockingRecv error: recv got EOF from 127.22.157.194:36521 (error 108)
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":569,"threads_started":1}
Sep 28 15:51:41 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:41 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:41.255546 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:41.247400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:52677)
0928 15:51:41.247890 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:41.247906 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:41.248025 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:41.248318 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:41.248321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:41.248683 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:41.248916 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:41.248926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.249929 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.249932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:41.250901 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:41.250912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.251565 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.251570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:41.251702 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:41.252308 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:41.252326 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:41.255304 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":270,"threads_started":1}
W20260928 15:51:41.256022 32550 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:41.247410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:52677 (local address 127.22.157.254:43797)
0928 15:51:41.247640 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:41.247650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:41.248024 (+ 374us) server_negotiation.cc:407] Connection header received
0928 15:51:41.248110 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:41.248113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:41.248183 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:41.248270 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:41.249055 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.249805 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.251039 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.251431 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.251527 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:41.255627 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:52677: BlockingRecv error: recv got EOF from 127.22.157.195:52677 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 15:51:41 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:41 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:41.265780 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:41.256542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:36131)
0928 15:51:41.257522 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:41.257540 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:41.257658 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:41.258050 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:41.258055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:41.258569 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:41.258744 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:41.258753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.260111 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.260116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:41.261069 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:41.261081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.261856 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.261860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:41.261993 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:41.262592 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:41.262611 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:41.265507 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":696,"threads_started":1}
W20260928 15:51:41.266132 32553 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:41.256670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:36131 (local address 127.22.157.254:43797)
0928 15:51:41.257329 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:41.257335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:41.257632 (+ 297us) server_negotiation.cc:407] Connection header received
0928 15:51:41.257772 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:41.257780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:41.257886 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:41.257977 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:41.258907 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.259972 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.261215 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.261703 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.261821 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:41.265818 (+ 3997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:36131: BlockingRecv error: recv got EOF from 127.22.157.193:36131 (error 108)
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":257,"threads_started":1}
Sep 28 15:51:41 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:41 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:41.792251 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:41.780610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:39905)
0928 15:51:41.781435 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:41.781477 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:41.781638 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:41.782083 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:41.782089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:41.782736 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:41.783026 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:41.783042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.784962 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.784968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:41.786553 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:41.786579 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.787659 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.787664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:41.787809 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:41.788519 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:41.788539 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:41.791984 (+ 3445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":345,"threads_started":1}
W20260928 15:51:41.792577 32555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:41.780795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:39905 (local address 127.22.157.254:43797)
0928 15:51:41.781588 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:41.781594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:41.781615 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:51:41.781793 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:41.781800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:41.781902 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:41.782022 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:41.783291 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.784767 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.786820 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:41.787457 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:41.787581 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:41.792236 (+ 4655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:39905: BlockingRecv error: recv got EOF from 127.22.157.194:39905 (error 108)
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":503,"threads_started":1}
Sep 28 15:51:42 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:42 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:42.265631 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:42.256610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:52111)
0928 15:51:42.257172 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:42.257188 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:42.257339 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:42.257687 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:42.257692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:42.258139 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:42.258314 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:42.258323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.259489 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.259495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:42.260412 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:42.260424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.261191 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.261197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:42.261375 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:42.262000 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:42.262018 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:42.265374 (+ 3356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":302,"threads_started":1}
W20260928 15:51:42.265798 32555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:42.256692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:52111 (local address 127.22.157.254:43797)
0928 15:51:42.256948 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:42.256957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:42.257270 (+ 313us) server_negotiation.cc:407] Connection header received
0928 15:51:42.257464 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:42.257470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:42.257542 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:42.257653 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:42.258463 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.259357 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.260555 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.261034 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.261164 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:42.265560 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:52111: BlockingRecv error: recv got EOF from 127.22.157.195:52111 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 15:51:42 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:42 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:42.274911 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:42.266708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:42883)
0928 15:51:42.267176 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:42.267193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:42.267297 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:42.267588 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:42.267591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:42.267926 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:42.268074 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:42.268081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.269072 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.269077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:42.270440 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:42.270455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.271136 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.271141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:42.271273 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:42.271819 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:42.271838 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:42.274696 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
W20260928 15:51:42.275185 32555 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:42.266767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:42883 (local address 127.22.157.254:43797)
0928 15:51:42.266997 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:42.267003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:42.267267 (+ 264us) server_negotiation.cc:407] Connection header received
0928 15:51:42.267395 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:42.267398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:42.267468 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:42.267543 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:42.268184 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.268934 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.270570 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.270968 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.271057 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:42.274900 (+ 3843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:42883: BlockingRecv error: recv got EOF from 127.22.157.193:42883 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 15:51:42 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:42 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:42.809764 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:42.793313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:57253)
0928 15:51:42.793906 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:42.793923 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:42.794032 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:42.794375 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:42.794378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:42.794752 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:42.794905 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:42.794913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.795993 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.795998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:42.797132 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:42.797147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.797935 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.797940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:42.798072 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:42.806458 (+ 8386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:42.806481 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:42.809537 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":343,"threads_started":1}
W20260928 15:51:42.810114 32559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:42.793364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:57253 (local address 127.22.157.254:43797)
0928 15:51:42.793941 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:42.793963 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:42.793982 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:51:42.794167 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:42.794171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:42.794246 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:42.794340 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:42.795041 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.795819 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.797280 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:42.797766 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:42.797870 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:42.809810 (+ 11940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:57253: BlockingRecv error: recv got EOF from 127.22.157.194:57253 (error 108)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":319,"threads_started":1}
Sep 28 15:51:43 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:43 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:43.276453 32560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:43.266734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:49137)
0928 15:51:43.267323 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:43.267340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:43.267523 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:43.268000 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:43.268006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:43.268582 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:43.268797 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:43.268808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.270047 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.270052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:43.271008 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:43.271022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.271978 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.271985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:43.272192 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:43.272954 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:43.272974 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:43.276035 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":332,"threads_started":1}
W20260928 15:51:43.276999 32559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:43.266851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:49137 (local address 127.22.157.254:43797)
0928 15:51:43.267117 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:43.267124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:43.267439 (+ 315us) server_negotiation.cc:407] Connection header received
0928 15:51:43.267704 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:43.267709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:43.267789 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:43.267914 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:43.268989 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.269863 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.271203 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.271769 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.271891 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:43.276350 (+ 4459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:49137: BlockingRecv error: recv got EOF from 127.22.157.195:49137 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 15:51:43 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:43 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:43.287169 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:43.275996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:59653)
0928 15:51:43.277743 (+ 1747us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:43.277760 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:43.277931 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:43.278312 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:43.278317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:43.278798 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:43.279019 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:43.279028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.280459 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.280466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:43.281559 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:43.281573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.282344 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.282351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:43.282532 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:43.283265 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:43.283287 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:43.286762 (+ 3475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1647,"thread_start_us":1445,"threads_started":1}
W20260928 15:51:43.287384 32559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:43.276089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:59653 (local address 127.22.157.254:43797)
0928 15:51:43.277556 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:43.277562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:43.277857 (+ 295us) server_negotiation.cc:407] Connection header received
0928 15:51:43.278076 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:43.278085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:43.278162 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:43.278271 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:43.279171 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.280309 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.281727 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.282171 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.282323 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:43.287074 (+ 4751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:59653: BlockingRecv error: recv got EOF from 127.22.157.193:59653 (error 108)
Metrics: {"server-negotiator.queue_time_us":1312,"thread_start_us":1192,"threads_started":1}
Sep 28 15:51:43 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:43 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:43.823097 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:43.811124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:53063)
0928 15:51:43.812398 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:43.812417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:43.812613 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:43.813043 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:43.813048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:43.813577 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:43.813767 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:43.813777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.814938 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.814945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:43.816052 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:43.816066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.817113 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.817120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:43.817313 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:43.818065 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:43.818110 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:43.822669 (+ 4559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1138,"thread_start_us":912,"threads_started":1}
W20260928 15:51:43.823252 32564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:43.811246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:53063 (local address 127.22.157.254:43797)
0928 15:51:43.811943 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:43.811955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:43.812529 (+ 574us) server_negotiation.cc:407] Connection header received
0928 15:51:43.812793 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:43.812798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:43.812875 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:43.813004 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:43.813939 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.814754 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.816217 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:43.816884 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:43.817094 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:43.822950 (+ 5856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:53063: BlockingRecv error: recv got EOF from 127.22.157.194:53063 (error 108)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":272,"threads_started":1}
Sep 28 15:51:44 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:44 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:44.286492 32565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:44.277814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:49559)
0928 15:51:44.278523 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:44.278558 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:44.278678 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:44.279015 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:44.279019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:44.279565 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:44.279738 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:44.279747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.280842 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.280846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:44.281835 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:44.281855 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.282543 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.282547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:44.282662 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:44.283255 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:44.283273 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:44.286237 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":293,"threads_started":1}
W20260928 15:51:44.286712 32564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:44.277909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:49559 (local address 127.22.157.254:43797)
0928 15:51:44.278302 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:44.278311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:44.278652 (+ 341us) server_negotiation.cc:407] Connection header received
0928 15:51:44.278778 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:44.278783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:44.278858 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:44.278969 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:44.279867 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.280650 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.281979 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.282381 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.282476 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:44.286467 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:49559: BlockingRecv error: recv got EOF from 127.22.157.195:49559 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 15:51:44 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:44 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:44.297479 32566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:44.288118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:50359)
0928 15:51:44.288686 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:44.288702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:44.288886 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:44.289262 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:44.289269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:44.289769 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:44.289932 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:44.289941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.291213 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.291219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:44.292427 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:44.292441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.293379 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.293385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:44.293554 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:44.294163 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:44.294179 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:44.297253 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":255,"threads_started":1}
W20260928 15:51:44.297737 32564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:44.288354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:50359 (local address 127.22.157.254:43797)
0928 15:51:44.288578 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:44.288584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:44.288830 (+ 246us) server_negotiation.cc:407] Connection header received
0928 15:51:44.289021 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:44.289026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:44.289114 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:44.289226 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:44.290071 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.291070 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.292597 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.293184 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.293333 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:44.297468 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:50359: BlockingRecv error: recv got EOF from 127.22.157.193:50359 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 15:51:44 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:44 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:44.833775 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:44.824110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:34613)
0928 15:51:44.824633 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:44.824650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:44.824828 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:44.825227 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:44.825230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:44.825706 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:44.825885 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:44.825897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.827240 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.827244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:44.828279 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:44.828293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.829272 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.829277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:44.829405 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:44.830130 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:44.830149 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:44.833524 (+ 3375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":261,"threads_started":1}
W20260928 15:51:44.834100 32568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:44.824180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:34613 (local address 127.22.157.254:43797)
0928 15:51:44.824912 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:44.824916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:44.824931 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:51:44.824975 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:44.824978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:44.825046 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:44.825143 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:44.826114 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.827062 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.828452 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:44.829052 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:44.829159 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:44.833793 (+ 4634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:34613: BlockingRecv error: recv got EOF from 127.22.157.194:34613 (error 108)
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":538,"threads_started":1}
Sep 28 15:51:45 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:45 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:45.296844 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:45.287710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:51719)
0928 15:51:45.288273 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:45.288288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:45.288453 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:45.288860 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:45.288865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:45.289293 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:45.289510 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:45.289523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.290678 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.290684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:45.291696 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:45.291708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.292455 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.292462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:45.292619 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:45.293405 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:45.293434 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:45.296525 (+ 3091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":301,"threads_started":1}
W20260928 15:51:45.297030 32568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:45.287807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:51719 (local address 127.22.157.254:43797)
0928 15:51:45.288058 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:45.288067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:45.288380 (+ 313us) server_negotiation.cc:407] Connection header received
0928 15:51:45.288601 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:45.288611 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:45.288674 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:45.288834 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:45.289672 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.290534 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.291845 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.292263 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.292391 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:45.296801 (+ 4410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:51719: BlockingRecv error: recv got EOF from 127.22.157.195:51719 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 28 15:51:45 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:45 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:45.306808 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:45.298279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:54813)
0928 15:51:45.298771 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:45.298786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:45.298932 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:45.299266 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:45.299271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:45.299675 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:45.299856 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:45.299893 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.300974 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.300982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:45.301930 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:45.301942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.302671 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.302677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:45.302848 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:45.303352 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:45.303369 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:45.306563 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":283,"threads_started":1}
W20260928 15:51:45.307425 32568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:45.298354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:54813 (local address 127.22.157.254:43797)
0928 15:51:45.298550 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:45.298559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:45.298865 (+ 306us) server_negotiation.cc:407] Connection header received
0928 15:51:45.299049 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:45.299056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:45.299132 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:45.299234 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:45.300028 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.300808 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.302088 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.302503 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.302639 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:45.307012 (+ 4373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:54813: BlockingRecv error: recv got EOF from 127.22.157.193:54813 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 15:51:45 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:45 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:45.843228 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:45.834740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:60859)
0928 15:51:45.835394 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:45.835421 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:45.835566 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:45.835901 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:45.835905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:45.836321 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:45.836486 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:45.836494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.837595 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.837600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:45.838557 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:45.838570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.839231 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.839235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:45.839361 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:45.839926 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:45.839943 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:45.842873 (+ 2930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":487,"threads_started":1}
W20260928 15:51:45.843395 32572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:45.834986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:60859 (local address 127.22.157.254:43797)
0928 15:51:45.835642 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:45.835646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:45.835660 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:51:45.835703 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:45.835706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:45.835763 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:45.835847 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:45.836641 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.837436 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.838683 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:45.839086 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:45.839175 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:45.843143 (+ 3968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:60859: BlockingRecv error: recv got EOF from 127.22.157.194:60859 (error 108)
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":459,"threads_started":1}
Sep 28 15:51:46 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:46 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:46.306551 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:46.297745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:44467)
0928 15:51:46.298293 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:46.298308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:46.298421 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:46.298716 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:46.298719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:46.299145 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:46.299344 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:46.299359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.300446 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.300451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:46.301530 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:46.301544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.302265 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.302270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:46.302395 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:46.302997 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:46.303023 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:46.306191 (+ 3168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":317,"threads_started":1}
W20260928 15:51:46.306699 32572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:46.297827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:44467 (local address 127.22.157.254:43797)
0928 15:51:46.298028 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:46.298035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:46.298378 (+ 343us) server_negotiation.cc:407] Connection header received
0928 15:51:46.298517 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:46.298521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:46.298585 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:46.298674 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:46.299478 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.300285 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.301715 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.302125 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.302221 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:46.306407 (+ 4186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:44467: BlockingRecv error: recv got EOF from 127.22.157.195:44467 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 15:51:46 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:46 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:46.316689 32574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:46.307665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:35725)
0928 15:51:46.308154 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:46.308175 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:46.308288 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:46.308617 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:46.308620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:46.309055 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:46.309201 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:46.309209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.310244 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.310248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:46.311241 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:46.311254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.311969 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.311973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:46.312105 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:46.312832 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:46.312859 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:46.316322 (+ 3463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":278,"threads_started":1}
W20260928 15:51:46.316921 32572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:46.307794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:35725 (local address 127.22.157.254:43797)
0928 15:51:46.307984 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:46.307989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:46.308256 (+ 267us) server_negotiation.cc:407] Connection header received
0928 15:51:46.308390 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:46.308394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:46.308474 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:46.308559 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:46.309351 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.310107 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.311371 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.311797 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.311890 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:46.316588 (+ 4698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:35725: BlockingRecv error: recv got EOF from 127.22.157.193:35725 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 15:51:46 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:46 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:46.853466 32575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:46.844456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:32821)
0928 15:51:46.845003 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:46.845023 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:46.845141 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:46.845738 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:46.845742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:46.846122 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:46.846275 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:46.846282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.847556 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.847561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:46.848502 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:46.848526 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.849361 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.849368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:46.849558 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:46.850155 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:46.850173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:46.853226 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":336,"threads_started":1}
W20260928 15:51:46.853786 32576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:46.844559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:32821 (local address 127.22.157.254:43797)
0928 15:51:46.845425 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:46.845430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:46.845447 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:51:46.845506 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:46.845509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:46.845583 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:46.845677 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:46.846460 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.847363 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.848663 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:46.849196 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:46.849341 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:46.853497 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:32821: BlockingRecv error: recv got EOF from 127.22.157.194:32821 (error 108)
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":580,"threads_started":1}
Sep 28 15:51:47 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:47 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:47.316530 32577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:47.307853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:56171)
0928 15:51:47.308370 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:47.308386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:47.308498 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:47.308848 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:47.308852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:47.309221 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:47.309369 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:47.309377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.310432 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.310436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:47.311432 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:47.311445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.312212 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.312217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:47.312350 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:47.313012 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:47.313030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:47.316143 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":287,"threads_started":1}
W20260928 15:51:47.316769 32576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:47.308048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:56171 (local address 127.22.157.254:43797)
0928 15:51:47.308329 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:47.308336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:47.308463 (+ 127us) server_negotiation.cc:407] Connection header received
0928 15:51:47.308611 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:47.308615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:47.308679 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:47.308791 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:47.309543 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.310296 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.311602 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.312035 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.312136 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:47.316446 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:56171: BlockingRecv error: recv got EOF from 127.22.157.195:56171 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Sep 28 15:51:47 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:47 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:47.329751 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:47.317844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:44647)
0928 15:51:47.321364 (+ 3520us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:47.321383 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:47.321567 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:47.322023 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:47.322028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:47.322512 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:47.322712 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:47.322721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.323766 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.323774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:47.324706 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:47.324766 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.325526 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.325533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:47.325719 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:47.326311 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:47.326328 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:47.329441 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3396,"thread_start_us":3216,"threads_started":1}
W20260928 15:51:47.330098 32576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:47.318012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:44647 (local address 127.22.157.254:43797)
0928 15:51:47.318211 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:47.318217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:47.321497 (+ 3280us) server_negotiation.cc:407] Connection header received
0928 15:51:47.321768 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:47.321772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:47.321863 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:47.321985 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:47.322871 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.323606 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.324903 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.325354 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.325490 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:47.329738 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:44647: BlockingRecv error: recv got EOF from 127.22.157.193:44647 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 15:51:47 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:47 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:47.870623 32579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:47.854604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:43039)
0928 15:51:47.855117 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:47.855135 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:47.855252 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:47.855787 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:47.855790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:47.856218 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:47.856412 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:47.856421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.857648 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.857653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:47.858570 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:47.858584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.859505 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.859510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:47.859717 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:47.866711 (+ 6994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:47.866738 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:47.870268 (+ 3530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":259,"threads_started":1}
W20260928 15:51:47.870860 32580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:47.854760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:43039 (local address 127.22.157.254:43797)
0928 15:51:47.855445 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:47.855450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:47.855511 (+ 61us) server_negotiation.cc:407] Connection header received
0928 15:51:47.855568 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:47.855571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:47.855645 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:47.855738 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:47.856619 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.857508 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.858723 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:47.859182 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:47.859283 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:47.870538 (+ 11255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:43039: BlockingRecv error: recv got EOF from 127.22.157.194:43039 (error 108)
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":441,"threads_started":1}
Sep 28 15:51:48 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:48 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:48.326408 32581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:48.317589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:59295)
0928 15:51:48.318195 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:48.318212 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:48.318329 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:48.318641 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:48.318645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:48.319072 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:48.319247 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:48.319256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.320317 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.320321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:48.321337 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:48.321350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.322066 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.322071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:48.322205 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:48.322839 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:48.322856 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:48.326056 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":326,"threads_started":1}
W20260928 15:51:48.326550 32580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:48.317704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:59295 (local address 127.22.157.254:43797)
0928 15:51:48.317883 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:48.317887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:48.318290 (+ 403us) server_negotiation.cc:407] Connection header received
0928 15:51:48.318433 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:48.318436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:48.318499 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:48.318589 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:48.319385 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.320183 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.321479 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.321879 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.321979 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:48.326321 (+ 4342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:59295: BlockingRecv error: recv got EOF from 127.22.157.195:59295 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 15:51:48 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:48 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:48.339395 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:48.330826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:54933)
0928 15:51:48.331338 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:48.331355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:48.331473 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:48.331808 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:48.331812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:48.332209 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:48.332363 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:48.332372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.333456 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.333460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:48.334429 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:48.334442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.335118 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.335123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:48.335255 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:48.335838 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:48.335856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:48.339107 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":305,"threads_started":1}
W20260928 15:51:48.339874 32580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:48.330925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:54933 (local address 127.22.157.254:43797)
0928 15:51:48.331213 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:48.331219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:48.331437 (+ 218us) server_negotiation.cc:407] Connection header received
0928 15:51:48.331574 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:48.331578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:48.331661 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:48.331742 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:48.332492 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.333299 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.334569 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.334981 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.335077 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:48.339527 (+ 4450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:54933: BlockingRecv error: recv got EOF from 127.22.157.193:54933 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 15:51:48 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:48 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:48.880407 32583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:48.871713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:44453)
0928 15:51:48.872351 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:48.872368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:48.872471 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:48.872933 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:48.872937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:48.873428 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:48.873642 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:48.873654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.874763 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.874768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:48.875714 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:48.875727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.876558 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.876562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:48.876742 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:48.877329 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:48.877347 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:48.880175 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":287,"threads_started":1}
W20260928 15:51:48.880676 32584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:48.871804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:44453 (local address 127.22.157.254:43797)
0928 15:51:48.872557 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:48.872565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:48.872585 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:51:48.872645 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:48.872649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:48.872716 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:48.872863 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:48.873788 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.874612 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.875849 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:48.876424 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:48.876520 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:48.880408 (+ 3888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:44453: BlockingRecv error: recv got EOF from 127.22.157.194:44453 (error 108)
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":534,"threads_started":1}
Sep 28 15:51:49 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:49 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:49.336045 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:49.327610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:38249)
0928 15:51:49.328103 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:49.328119 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:49.328243 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:49.328634 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:49.328638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:49.329105 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:49.329316 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:49.329332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.330686 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.330690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:49.331628 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:49.331640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.332260 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.332264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:49.332388 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:49.332992 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:49.333010 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:49.335792 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":292,"threads_started":1}
W20260928 15:51:49.336218 32584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:49.327674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:38249 (local address 127.22.157.254:43797)
0928 15:51:49.327914 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:49.327924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:49.328243 (+ 319us) server_negotiation.cc:407] Connection header received
0928 15:51:49.328356 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:49.328361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:49.328441 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:49.328557 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:49.329471 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.330553 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.331751 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.332140 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.332232 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:49.335982 (+ 3750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:38249: BlockingRecv error: recv got EOF from 127.22.157.195:38249 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 15:51:49 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:49 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:49.348974 32586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:49.340389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:52549)
0928 15:51:49.341030 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:49.341047 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:49.341157 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:49.341517 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:49.341523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:49.342067 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:49.342237 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:49.342246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.343348 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.343354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:49.344279 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:49.344291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.345106 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.345112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:49.345291 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:49.345906 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:49.345924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:49.348602 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":347,"threads_started":1}
W20260928 15:51:49.349098 32584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:49.340473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:52549 (local address 127.22.157.254:43797)
0928 15:51:49.340659 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:49.340665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:49.341118 (+ 453us) server_negotiation.cc:407] Connection header received
0928 15:51:49.341253 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:49.341258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:49.341356 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:49.341468 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:49.342378 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.343207 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.344433 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.344921 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.345047 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:49.348827 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:52549: BlockingRecv error: recv got EOF from 127.22.157.193:52549 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 15:51:49 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:49 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:49.890410 32587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:49.881358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:57755)
0928 15:51:49.881928 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:49.881947 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:49.882062 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:49.882421 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:49.882424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:49.882819 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:49.883025 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:49.883034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.884048 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.884052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:49.885283 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:49.885298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.886037 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.886041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:49.886169 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:49.886922 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:49.886943 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:49.890129 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":281,"threads_started":1}
W20260928 15:51:49.890648 32588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:49.881619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:57755 (local address 127.22.157.254:43797)
0928 15:51:49.882150 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:49.882154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:49.882168 (+ 14us) server_negotiation.cc:407] Connection header received
0928 15:51:49.882212 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:49.882215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:49.882281 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:49.882369 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:49.883164 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.883873 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.885426 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:49.885858 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:49.885953 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:49.890363 (+ 4410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:57755: BlockingRecv error: recv got EOF from 127.22.157.194:57755 (error 108)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":325,"threads_started":1}
Sep 28 15:51:50 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:50 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:50.347386 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:50.337439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:48113)
0928 15:51:50.338126 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:50.338144 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:50.338283 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:50.338798 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:50.338806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:50.339299 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:50.339484 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:50.339494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.340975 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.340981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:50.341956 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:50.341969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.342917 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.342925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:50.343072 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:50.343898 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:50.343920 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:50.347113 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":359,"threads_started":1}
W20260928 15:51:50.347755 32588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:50.337564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:48113 (local address 127.22.157.254:43797)
0928 15:51:50.337790 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:50.337794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:50.338239 (+ 445us) server_negotiation.cc:407] Connection header received
0928 15:51:50.338528 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:50.338533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:50.338603 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:50.338705 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:50.339669 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.340691 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.342130 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.342652 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.342764 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:50.347397 (+ 4633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:48113: BlockingRecv error: recv got EOF from 127.22.157.195:48113 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 28 15:51:50 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:50 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:50.358743 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:50.350118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:50001)
0928 15:51:50.350646 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:50.350670 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:50.350794 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:50.351187 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:50.351192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:50.351646 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:50.351824 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:50.351834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.352912 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.352917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:50.353843 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:50.353855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.354626 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.354632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:50.354818 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:50.355478 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:50.355496 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:50.358478 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":304,"threads_started":1}
W20260928 15:51:50.359158 32588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:50.350227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:50001 (local address 127.22.157.254:43797)
0928 15:51:50.350505 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:50.350509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:50.350775 (+ 266us) server_negotiation.cc:407] Connection header received
0928 15:51:50.350902 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:50.350906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:50.350987 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:50.351098 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:50.351979 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.352711 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.353992 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.354461 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.354608 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:50.358718 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:50001: BlockingRecv error: recv got EOF from 127.22.157.193:50001 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
Sep 28 15:51:50 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:50 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:50.900642 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:50.891429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:35785)
0928 15:51:50.891959 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:50.891975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:50.892088 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:50.892482 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:50.892486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:50.892949 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:50.893131 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:50.893140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.894301 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.894305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:50.895200 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:50.895210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.896135 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.896142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:50.896320 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:50.897124 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:50.897149 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:50.900370 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":293,"threads_started":1}
W20260928 15:51:50.901022 32592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:50.891551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:35785 (local address 127.22.157.254:43797)
0928 15:51:50.892109 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:50.892114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:50.892142 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:51:50.892209 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:50.892213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:50.892288 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:50.892394 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:50.893275 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.894170 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.895339 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:50.895911 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:50.896059 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:50.900645 (+ 4586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:35785: BlockingRecv error: recv got EOF from 127.22.157.194:35785 (error 108)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":305,"threads_started":1}
Sep 28 15:51:51 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:51 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.195@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:51.357501 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:51.348636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.195:59293)
0928 15:51:51.349211 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:51.349229 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:51.349376 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:51.349813 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:51.349818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:51.350263 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:51.350442 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:51.350452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.351482 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.351488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:51.352454 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:51.352466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.353262 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.353268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:51.353439 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:51.354036 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:51.354053 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:51.357098 (+ 3045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":318,"threads_started":1}
W20260928 15:51:51.357621 32592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:51.348736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:59293 (local address 127.22.157.254:43797)
0928 15:51:51.348992 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:51.349000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:51.349361 (+ 361us) server_negotiation.cc:407] Connection header received
0928 15:51:51.349519 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:51.349524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:51.349615 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:51.349741 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:51.350599 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.351331 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.352601 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.353108 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.353239 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:51.357349 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.195:59293: BlockingRecv error: recv got EOF from 127.22.157.195:59293 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 15:51:51 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:51 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.193@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:51.368891 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:51.359897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.193:35909)
0928 15:51:51.360443 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:51.360462 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:51.360572 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:51.360988 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:51.360993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:51.361445 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:51.361631 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:51.361641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.362726 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.362734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:51.364182 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:51.364197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.364943 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.364948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:51.365151 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:51.365779 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:51.365797 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:51.368572 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":319,"threads_started":1}
W20260928 15:51:51.369243 32592 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:51.360023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:35909 (local address 127.22.157.254:43797)
0928 15:51:51.360212 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:51.360216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:51.360540 (+ 324us) server_negotiation.cc:407] Connection header received
0928 15:51:51.360671 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:51.360674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:51.360803 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:51.360886 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:51.361770 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.362571 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.364312 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.364719 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.364876 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:51.368816 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.193:35909: BlockingRecv error: recv got EOF from 127.22.157.193:35909 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 15:51:51 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 15:51:51 dist-test-slave-1vw0 krb5kdc[31520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.157.194@KRBTEST.COM for oryx/127.22.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 15:51:51.910727 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:51.901829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43797 (local address 127.22.157.194:41669)
0928 15:51:51.902457 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:51.902475 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:51.902588 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:51.902990 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:51.902994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:51.903550 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:51.903764 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:51.903777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.905123 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.905127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:51.906043 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:51.906055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.906733 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.906738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:51.906863 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:51.907524 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:51.907542 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:51.910354 (+ 2812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.157.254:43797: Server oryx/127.22.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":369,"threads_started":1}
W20260928 15:51:51.910863 32596 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:51.901914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:41669 (local address 127.22.157.254:43797)
0928 15:51:51.902521 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:51.902527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:51.902552 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:51:51.902720 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:51.902726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:51.902805 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:51.902937 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:51.903941 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.904983 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.906169 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:51.906609 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:51.906703 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:51.910593 (+ 3890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.157.194:41669: BlockingRecv error: recv got EOF from 127.22.157.194:41669 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":290,"threads_started":1}
I20260928 15:51:52.188804 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 32114
I20260928 15:51:52.202054 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:52.190140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.22.157.193:45907)
0928 15:51:52.190761 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:52.190768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:52.190839 (+ 71us) server_negotiation.cc:407] Connection header received
0928 15:51:52.191062 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:52.191070 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:52.191298 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:52.191465 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:52.193099 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:52.194169 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:52.195327 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:52.195878 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:52.195972 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:52.198722 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:52.198730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:52.198745 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:52.198779 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:52.201442 (+ 2663us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.157.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 15:51:52.201804 (+ 362us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44684: Request ticket server kudu/127.22.157.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":338,"threads_started":1}
W20260928 15:51:52.202183 32598 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44684: Request ticket server kudu/127.22.157.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 15:51:52.202278 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 32114: Not authorized: Client connection negotiation failed: client connection to 127.22.157.193:45907: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.157.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 15:51:52.202535 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 32114
I20260928 15:51:52.220894 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 32267
I20260928 15:51:52.235535 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:52.222339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.22.157.194:44923)
0928 15:51:52.223057 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:52.223065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:52.223104 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:51:52.223171 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:52.223178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:52.223412 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:52.223607 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:52.225143 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:52.226363 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:52.227887 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:52.228586 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:52.228817 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:52.231664 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:52.231674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:52.231691 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:52.231736 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:52.234938 (+ 3202us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.157.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 15:51:52.235266 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33792: Request ticket server kudu/127.22.157.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":394,"threads_started":1}
W20260928 15:51:52.235644 32599 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33792: Request ticket server kudu/127.22.157.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 15:51:52.235947 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 32267: Not authorized: Client connection negotiation failed: client connection to 127.22.157.194:44923: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.157.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 15:51:52.236083 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 32267
I20260928 15:51:52.253428 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 32401
I20260928 15:51:52.266947 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:52.254508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.22.157.195:36459)
0928 15:51:52.255172 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:52.255179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:52.255208 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:51:52.255271 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:52.255277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:52.255580 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:52.255752 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:52.257376 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:52.258531 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:52.259950 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:52.260593 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:52.260703 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:52.263488 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:52.263497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:52.263512 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:52.263549 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:52.266391 (+ 2842us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.157.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 15:51:52.266694 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37212: Request ticket server kudu/127.22.157.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":332,"threads_started":1}
W20260928 15:51:52.267089 32600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37212: Request ticket server kudu/127.22.157.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 15:51:52.267138 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 32401: Not authorized: Client connection negotiation failed: client connection to 127.22.157.195:36459: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.157.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 15:51:52.267343 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 32401
I20260928 15:51:52.284168 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 32045
I20260928 15:51:52.285671 32107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:52.406962 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 32045
2026-09-28T15:51:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21490 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T15:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:52Z Disabled control of system clock
I20260928 15:51:52.493353 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:37921
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:37921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 15:51:52.838485 32605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:52.838833 32605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:52.838920 32605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:52.846287 32605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:52.846415 32605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:52.846454 32605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:52.846513 32605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:52.846562 32605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:52.856127 32605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:37921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:52.857631 32605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:52.859489 32605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:52.873347 32611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:52.873474 32610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:52.873451 32613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:52.873708 32605 server_base.cc:1034] running on GCE node
I20260928 15:51:52.874850 32605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:52.876396 32605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:52.877669 32605 hybrid_clock.cc:648] HybridClock initialized: now 1790610712877592 us; error 73 us; skew 500 ppm
I20260928 15:51:52.878057 32605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:52.886562 32605 webserver.cc:460] Webserver started at http://127.22.157.254:39201/ using document root <none> and password file <none>
I20260928 15:51:52.887408 32605 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:52.887565 32605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:52.887928 32605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:52.891233 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "31f856ff953d4a0c8825a6875eb68d09"
format_stamp: "Formatted at 2026-09-28 15:51:52 on dist-test-slave-1vw0"
server_key: "b1c187d2e4222066ee6dc58e1c842e60"
server_key_iv: "8229670442aede0455de7e6a26ffcbaa"
server_key_version: "encryptionkey@0"
I20260928 15:51:52.892145 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "31f856ff953d4a0c8825a6875eb68d09"
format_stamp: "Formatted at 2026-09-28 15:51:52 on dist-test-slave-1vw0"
server_key: "b1c187d2e4222066ee6dc58e1c842e60"
server_key_iv: "8229670442aede0455de7e6a26ffcbaa"
server_key_version: "encryptionkey@0"
I20260928 15:51:52.898108 32605 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 15:51:52.902029 32619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:52.903549 32605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:51:52.903800 32605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "31f856ff953d4a0c8825a6875eb68d09"
format_stamp: "Formatted at 2026-09-28 15:51:52 on dist-test-slave-1vw0"
server_key: "b1c187d2e4222066ee6dc58e1c842e60"
server_key_iv: "8229670442aede0455de7e6a26ffcbaa"
server_key_version: "encryptionkey@0"
I20260928 15:51:52.904065 32605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:52.924095 32605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:52.925187 32605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:52.925513 32605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:52.945580 32605 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:37921
I20260928 15:51:52.945587 32670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:37921 every 8 connection(s)
I20260928 15:51:52.947085 32605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:52.949646 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 32605
I20260928 15:51:52.949851 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:52.950671 23159 external_mini_cluster.cc:1428] Setting key 9bebadf8ce080a4cc447efa436ae044a
I20260928 15:51:52.951918 32671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:52.964632 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:52.952885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.22.157.254:37921)
0928 15:51:52.953924 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:52.953936 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:52.953979 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:51:52.955626 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:52.955650 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:52.956162 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:52.956614 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:52.957277 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:52.958653 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:52.960046 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:52.960828 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:52.961022 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:52.962271 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:52.962279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:52.962293 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:52.962885 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:52.962962 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:52.962990 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:52.963126 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:52.963359 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 15:51:52.963666 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":365,"threads_started":1}
I20260928 15:51:52.965122 32671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09: Bootstrap starting.
I20260928 15:51:52.968420 32671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:52.969782 32671 log.cc:824] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:52.972662 32671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09: No bootstrap required, opened a new log
I20260928 15:51:52.980340 32671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f856ff953d4a0c8825a6875eb68d09" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 37921 } }
I20260928 15:51:52.980715 32671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:52.980870 32671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f856ff953d4a0c8825a6875eb68d09, State: Initialized, Role: FOLLOWER
I20260928 15:51:52.981465 32671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f856ff953d4a0c8825a6875eb68d09" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 37921 } }
I20260928 15:51:52.981673 32671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:52.981775 32671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:52.981907 32671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:52.984266 32671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f856ff953d4a0c8825a6875eb68d09" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 37921 } }
I20260928 15:51:52.984906 32671 leader_election.cc:302] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31f856ff953d4a0c8825a6875eb68d09; no voters:
I20260928 15:51:52.985495 32671 leader_election.cc:288] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:52.985719 32676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:52.986493 32676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [term 1 LEADER]: Becoming Leader. State: Replica: 31f856ff953d4a0c8825a6875eb68d09, State: Running, Role: LEADER
I20260928 15:51:52.987082 32676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f856ff953d4a0c8825a6875eb68d09" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 37921 } }
I20260928 15:51:52.987860 32671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:52.990123 32678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f856ff953d4a0c8825a6875eb68d09. Latest consensus state: current_term: 1 leader_uuid: "31f856ff953d4a0c8825a6875eb68d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f856ff953d4a0c8825a6875eb68d09" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 37921 } } }
I20260928 15:51:52.991055 32678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:52.993270 32677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31f856ff953d4a0c8825a6875eb68d09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f856ff953d4a0c8825a6875eb68d09" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 37921 } } }
I20260928 15:51:52.993687 32677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:52.993989 32686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:52.999145 32686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:53.009282 32686 catalog_manager.cc:1347] Generated new cluster ID: c01302688c834ddcadeb6ad49bb656d0
I20260928 15:51:53.009519 32686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:53.030216 32686 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:53.031747 32686 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:53.044843 32686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31f856ff953d4a0c8825a6875eb68d09: Generated new TSK 0
I20260928 15:51:53.045814 32686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:51:53.052100 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:37921
--builtin_ntp_servers=127.22.157.212:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 15:51:53.399045 32696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:53.399348 32696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:53.399494 32696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:53.407045 32696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:53.407159 32696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:53.407335 32696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:53.417471 32696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:37921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:53.418893 32696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:53.420780 32696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:53.435640 32701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:53.435595 32704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:53.435636 32702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:53.436589 32696 server_base.cc:1034] running on GCE node
I20260928 15:51:53.437361 32696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:53.438477 32696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:53.439697 32696 hybrid_clock.cc:648] HybridClock initialized: now 1790610713439643 us; error 46 us; skew 500 ppm
I20260928 15:51:53.440070 32696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:53.442674 32696 webserver.cc:460] Webserver started at http://127.22.157.193:40225/ using document root <none> and password file <none>
I20260928 15:51:53.443440 32696 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:53.443593 32696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:53.444010 32696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:53.447222 32696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "26f3129c9a9b4ad4b8560279fcdb8146"
format_stamp: "Formatted at 2026-09-28 15:51:53 on dist-test-slave-1vw0"
server_key: "3d9ac941168acfd9e815bc937705616e"
server_key_iv: "f06b91fc8bf55edcddf7c44f5a98e9d4"
server_key_version: "encryptionkey@0"
I20260928 15:51:53.448130 32696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "26f3129c9a9b4ad4b8560279fcdb8146"
format_stamp: "Formatted at 2026-09-28 15:51:53 on dist-test-slave-1vw0"
server_key: "3d9ac941168acfd9e815bc937705616e"
server_key_iv: "f06b91fc8bf55edcddf7c44f5a98e9d4"
server_key_version: "encryptionkey@0"
I20260928 15:51:53.453835 32696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 15:51:53.457816 32710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:53.459496 32696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:53.459728 32696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "26f3129c9a9b4ad4b8560279fcdb8146"
format_stamp: "Formatted at 2026-09-28 15:51:53 on dist-test-slave-1vw0"
server_key: "3d9ac941168acfd9e815bc937705616e"
server_key_iv: "f06b91fc8bf55edcddf7c44f5a98e9d4"
server_key_version: "encryptionkey@0"
I20260928 15:51:53.460005 32696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:53.478055 32696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:53.479038 32696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:53.479373 32696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:53.480433 32696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:53.482154 32696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:53.482244 32696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:53.482409 32696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:53.482507 32696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:53.517510 32696 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:41409
I20260928 15:51:53.517554 354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:41409 every 8 connection(s)
I20260928 15:51:53.518913 32696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:53.528702 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 32696
I20260928 15:51:53.528986 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:51:53.529650 23159 external_mini_cluster.cc:1428] Setting key 17b0e36b3ca0e5f3c23f96b95d2f4b44
I20260928 15:51:53.532691 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:37921
--builtin_ntp_servers=127.22.157.212:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 15:51:53.533617 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:53.521577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:37245 (local address 127.22.157.254:37921)
0928 15:51:53.522039 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:53.522044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:53.523298 (+ 1254us) server_negotiation.cc:407] Connection header received
0928 15:51:53.525210 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:53.525216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:53.525314 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:53.525435 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:53.526603 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:53.527391 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:53.529264 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:53.529697 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:53.529869 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:53.531618 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:53.531623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:53.531628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:53.531649 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:53.531701 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:53.531718 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:53.531820 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:53.533156 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0928 15:51:53.533268 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":231,"threads_started":1}
I20260928 15:51:53.534818 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:53.522213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:37921 (local address 127.22.157.193:37245)
0928 15:51:53.523125 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:53.523166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:53.524927 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:53.525697 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:53.525706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:53.525799 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:53.526413 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:53.526440 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:53.527582 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:53.527586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:53.529105 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:53.529122 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:53.530044 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:53.530054 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:53.530435 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:53.531197 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:53.531200 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:53.531244 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:53.531331 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:53.532849 (+ 1518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:53.532864 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:53.532880 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 15:51:53.533145 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 15:51:53.533614 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":275,"threads_started":1}
I20260928 15:51:53.536595 356 heartbeater.cc:347] Connected to a master server at 127.22.157.254:37921
I20260928 15:51:53.536988 356 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:53.537874 356 heartbeater.cc:511] Master 127.22.157.254:37921 requested a full tablet report, sending...
I20260928 15:51:53.540376 32636 ts_manager.cc:194] Registered new tserver with Master: 26f3129c9a9b4ad4b8560279fcdb8146 (127.22.157.193:41409)
I20260928 15:51:53.542052 32636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.193:37245
W20260928 15:51:53.888543 360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:53.888959 360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:53.889066 360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:53.896508 360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:53.896662 360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:53.896859 360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:51:53.906220 360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:37921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:53.907577 360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:53.909592 360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:53.923210 368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:53.923126 365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:53.923223 366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:53.926446 360 server_base.cc:1034] running on GCE node
I20260928 15:51:53.927131 360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:53.928128 360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:53.929354 360 hybrid_clock.cc:648] HybridClock initialized: now 1790610713929311 us; error 44 us; skew 500 ppm
I20260928 15:51:53.929687 360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:53.932125 360 webserver.cc:460] Webserver started at http://127.22.157.194:38735/ using document root <none> and password file <none>
I20260928 15:51:53.932929 360 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:53.933070 360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:53.933409 360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:53.936385 360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "e0d534dc4ede4bf6a0d7984de9955e8a"
format_stamp: "Formatted at 2026-09-28 15:51:53 on dist-test-slave-1vw0"
server_key: "e2a4601647fb28315e0e091eeee8bcb4"
server_key_iv: "30fa7055d59f69700081a1514afdda4f"
server_key_version: "encryptionkey@0"
I20260928 15:51:53.937250 360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "e0d534dc4ede4bf6a0d7984de9955e8a"
format_stamp: "Formatted at 2026-09-28 15:51:53 on dist-test-slave-1vw0"
server_key: "e2a4601647fb28315e0e091eeee8bcb4"
server_key_iv: "30fa7055d59f69700081a1514afdda4f"
server_key_version: "encryptionkey@0"
I20260928 15:51:53.942544 360 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:51:53.946400 374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:53.947965 360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:53.948179 360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "e0d534dc4ede4bf6a0d7984de9955e8a"
format_stamp: "Formatted at 2026-09-28 15:51:53 on dist-test-slave-1vw0"
server_key: "e2a4601647fb28315e0e091eeee8bcb4"
server_key_iv: "30fa7055d59f69700081a1514afdda4f"
server_key_version: "encryptionkey@0"
I20260928 15:51:53.948439 360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:53.968250 360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:53.969286 360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:53.969617 360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:53.970624 360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:53.972129 360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:53.972261 360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:53.972388 360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:53.972476 360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:54.006886 360 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:35931
I20260928 15:51:54.006932 486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:35931 every 8 connection(s)
I20260928 15:51:54.008412 360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:51:54.011332 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 360
I20260928 15:51:54.011502 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:51:54.011967 23159 external_mini_cluster.cc:1428] Setting key c88e4a3c6dd1021b74242334c4c2969e
I20260928 15:51:54.014854 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:37921
--builtin_ntp_servers=127.22.157.212:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 15:51:54.021632 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:54.011197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:48773 (local address 127.22.157.254:37921)
0928 15:51:54.011439 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:54.011447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:54.013141 (+ 1694us) server_negotiation.cc:407] Connection header received
0928 15:51:54.014943 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:54.014948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:54.015035 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:54.015163 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:54.016277 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.017107 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.018707 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.019111 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.019247 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:54.020646 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:54.020653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:54.020658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:54.020687 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:54.020765 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:54.020782 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:54.020887 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:54.021176 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 15:51:54.021282 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 15:51:54.022733 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:54.011817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:37921 (local address 127.22.157.194:48773)
0928 15:51:54.012971 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:54.013017 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:54.014666 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:54.015412 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:54.015421 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:54.015515 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:54.016107 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:54.016123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.017275 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.017279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:54.018545 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:54.018562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.019348 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.019355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:54.019593 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:54.020322 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:54.020325 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:54.020371 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:54.020450 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:54.020885 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:54.020897 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:54.020907 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:51:54.021144 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 15:51:54.021600 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":362,"threads_started":1}
I20260928 15:51:54.024302 487 heartbeater.cc:347] Connected to a master server at 127.22.157.254:37921
I20260928 15:51:54.024638 487 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:54.025460 487 heartbeater.cc:511] Master 127.22.157.254:37921 requested a full tablet report, sending...
I20260928 15:51:54.026890 32636 ts_manager.cc:194] Registered new tserver with Master: e0d534dc4ede4bf6a0d7984de9955e8a (127.22.157.194:35931)
I20260928 15:51:54.027729 32636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.194:48773
W20260928 15:51:54.354699 490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:54.355018 490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:54.355180 490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:54.362408 490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:54.362562 490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:54.362735 490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:51:54.372125 490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:37921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:54.373497 490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:54.375325 490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:54.388886 495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:54.388881 498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:54.388945 496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:54.390360 490 server_base.cc:1034] running on GCE node
I20260928 15:51:54.391214 490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:54.392333 490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:54.393590 490 hybrid_clock.cc:648] HybridClock initialized: now 1790610714393512 us; error 78 us; skew 500 ppm
I20260928 15:51:54.393994 490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:54.396402 490 webserver.cc:460] Webserver started at http://127.22.157.195:44853/ using document root <none> and password file <none>
I20260928 15:51:54.397239 490 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:54.397375 490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:54.397722 490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:54.400801 490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "add129f2432240e09d1055cb8c66f537"
format_stamp: "Formatted at 2026-09-28 15:51:54 on dist-test-slave-1vw0"
server_key: "eba024755861d2a276d66ae732233341"
server_key_iv: "946e5dfe26aea80a5a7ab33d0080ee24"
server_key_version: "encryptionkey@0"
I20260928 15:51:54.401644 490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "add129f2432240e09d1055cb8c66f537"
format_stamp: "Formatted at 2026-09-28 15:51:54 on dist-test-slave-1vw0"
server_key: "eba024755861d2a276d66ae732233341"
server_key_iv: "946e5dfe26aea80a5a7ab33d0080ee24"
server_key_version: "encryptionkey@0"
I20260928 15:51:54.407061 490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:51:54.410956 504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:54.412425 490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:51:54.412673 490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "add129f2432240e09d1055cb8c66f537"
format_stamp: "Formatted at 2026-09-28 15:51:54 on dist-test-slave-1vw0"
server_key: "eba024755861d2a276d66ae732233341"
server_key_iv: "946e5dfe26aea80a5a7ab33d0080ee24"
server_key_version: "encryptionkey@0"
I20260928 15:51:54.412966 490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:54.444897 490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:54.445922 490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:54.446264 490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:54.447333 490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:54.449040 490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:54.449182 490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:54.449312 490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:54.449399 490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:54.484745 490 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:44051
I20260928 15:51:54.484855 618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:44051 every 8 connection(s)
I20260928 15:51:54.486160 490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:51:54.490775 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 490
I20260928 15:51:54.490950 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:51:54.491489 23159 external_mini_cluster.cc:1428] Setting key c18a0e5f724bf8885cfc40cd1809196b
I20260928 15:51:54.500905 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:54.489027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:38651 (local address 127.22.157.254:37921)
0928 15:51:54.489240 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:54.489246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:54.491244 (+ 1998us) server_negotiation.cc:407] Connection header received
0928 15:51:54.493617 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:54.493622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:54.493713 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:54.493820 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:54.495354 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.496157 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.498346 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.498763 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.498860 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:54.500192 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:54.500197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:54.500200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:54.500219 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:54.500249 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:54.500261 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:54.500329 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:54.500532 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 15:51:54.500614 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:51:54.502012 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:54.489859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:37921 (local address 127.22.157.195:38651)
0928 15:51:54.491054 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:54.491104 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:54.493188 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:54.494163 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:54.494178 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:54.494315 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:54.495194 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:54.495216 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.496319 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.496323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:54.498165 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:54.498181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.498948 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.498953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:54.499163 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:54.499893 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:54.499896 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:54.499936 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:54.500019 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:54.500346 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:54.500353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:54.500359 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:51:54.500527 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 15:51:54.500930 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":353,"threads_started":1}
I20260928 15:51:54.503445 619 heartbeater.cc:347] Connected to a master server at 127.22.157.254:37921
I20260928 15:51:54.503784 619 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:54.504555 619 heartbeater.cc:511] Master 127.22.157.254:37921 requested a full tablet report, sending...
I20260928 15:51:54.506227 32636 ts_manager.cc:194] Registered new tserver with Master: add129f2432240e09d1055cb8c66f537 (127.22.157.195:44051)
I20260928 15:51:54.507165 32636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.195:38651
I20260928 15:51:54.507741 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:51:54.514539 358 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:54.512619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.22.157.254:37921)
0928 15:51:54.512872 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:54.512879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:54.513345 (+ 466us) server_negotiation.cc:407] Connection header received
0928 15:51:54.513560 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:54.513571 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:54.513646 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:54.513756 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:54.514242 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55840: BlockingRecv error: recv got EOF from 127.0.0.1:55840 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:51:54.515381 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 32696
I20260928 15:51:54.524019 628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:54.516239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.22.157.193:41409)
0928 15:51:54.516832 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:54.516840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:54.516865 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:51:54.516918 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:54.516924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:54.517241 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:54.517403 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:54.517781 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.518861 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.520662 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.521318 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.521416 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:54.522572 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:54.522580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:54.522593 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:54.523163 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:54.523209 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:54.523229 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:54.523366 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:54.523532 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 15:51:54.523663 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
I20260928 15:51:54.525273 350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:54.606572 356 heartbeater.cc:503] Master 127.22.157.254:37921 was elected leader, sending a full tablet report...
I20260928 15:51:54.607800 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 32696
W20260928 15:51:54.625597 32623 connection.cc:535] server connection from 127.22.157.193:37245 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:51:54.625849 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 360
I20260928 15:51:54.634589 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:54.626931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.22.157.194:35931)
0928 15:51:54.627439 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:54.627454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:54.627483 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:51:54.627546 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:54.627552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:54.627784 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:54.627963 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:54.628429 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.629570 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.630852 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.631465 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.631615 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:54.633032 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:54.633041 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:54.633056 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:54.633687 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:54.633737 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:54.633757 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:54.633894 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:54.634042 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 15:51:54.634222 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":242,"threads_started":1}
I20260928 15:51:54.636089 482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:54.720610 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 360
I20260928 15:51:54.736967 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 490
I20260928 15:51:54.745728 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:54.738163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.22.157.195:44051)
0928 15:51:54.738871 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:54.738877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:54.738909 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:51:54.738974 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:54.738980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:54.739197 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:54.739334 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:54.739758 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.740903 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.742343 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:54.742953 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:54.743063 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:54.744179 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:54.744186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:54.744201 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:54.744892 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:54.744951 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:54.744972 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:54.745064 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:54.745248 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 15:51:54.745375 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":400,"threads_started":1}
I20260928 15:51:54.747053 614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:54.832280 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 490
I20260928 15:51:54.848497 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 32605
I20260928 15:51:54.849869 32666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:54.954066 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 32605
2026-09-28T15:51:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2536 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T15:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:55Z Disabled control of system clock
I20260928 15:51:55.029398 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43325
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:43325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 15:51:55.380388 639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:55.380771 639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:55.380937 639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:55.388796 639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:55.388944 639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:55.389034 639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:55.389142 639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:55.389223 639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:55.399449 639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:43325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:55.400980 639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:55.402915 639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:55.415783 644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:55.415807 645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:55.415848 647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:55.416692 639 server_base.cc:1034] running on GCE node
I20260928 15:51:55.417608 639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:55.419123 639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:55.420370 639 hybrid_clock.cc:648] HybridClock initialized: now 1790610715420321 us; error 60 us; skew 500 ppm
I20260928 15:51:55.420825 639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:55.423295 639 webserver.cc:460] Webserver started at http://127.22.157.254:33367/ using document root <none> and password file <none>
I20260928 15:51:55.424114 639 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:55.424252 639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:55.424599 639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:55.427723 639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "3e0e640aac644f24bc0d63de1cb3d0d5"
format_stamp: "Formatted at 2026-09-28 15:51:55 on dist-test-slave-1vw0"
server_key: "4203bd8c686b2bfef591999bc65c1a0d"
server_key_iv: "23ed2fb8c69966e3ef5e6eec24ece8d4"
server_key_version: "encryptionkey@0"
I20260928 15:51:55.428573 639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "3e0e640aac644f24bc0d63de1cb3d0d5"
format_stamp: "Formatted at 2026-09-28 15:51:55 on dist-test-slave-1vw0"
server_key: "4203bd8c686b2bfef591999bc65c1a0d"
server_key_iv: "23ed2fb8c69966e3ef5e6eec24ece8d4"
server_key_version: "encryptionkey@0"
I20260928 15:51:55.434075 639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:51:55.437767 653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:55.439201 639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:51:55.439396 639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "3e0e640aac644f24bc0d63de1cb3d0d5"
format_stamp: "Formatted at 2026-09-28 15:51:55 on dist-test-slave-1vw0"
server_key: "4203bd8c686b2bfef591999bc65c1a0d"
server_key_iv: "23ed2fb8c69966e3ef5e6eec24ece8d4"
server_key_version: "encryptionkey@0"
I20260928 15:51:55.439640 639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:55.445832 639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:55.446664 639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:55.446983 639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:55.465899 639 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:43325
I20260928 15:51:55.465941 704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:43325 every 8 connection(s)
I20260928 15:51:55.467429 639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:55.471298 705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:55.474792 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 639
I20260928 15:51:55.475018 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:55.475690 23159 external_mini_cluster.cc:1428] Setting key 682997a6424101d4dfbbb3b1ec763027
I20260928 15:51:55.484135 705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5: Bootstrap starting.
I20260928 15:51:55.486358 708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:55.477896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.22.157.254:43325)
0928 15:51:55.478999 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:55.479017 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:55.479089 (+ 72us) server_negotiation.cc:407] Connection header received
0928 15:51:55.482100 (+ 3011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:55.482133 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:55.482706 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:55.483177 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:55.484231 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:55.484239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:55.484256 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:55.484275 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:55.484355 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:55.484378 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:55.484604 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:55.484764 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 15:51:55.485036 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":338,"threads_started":1}
I20260928 15:51:55.487910 705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:55.489279 705 log.cc:824] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:51:55.492244 705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5: No bootstrap required, opened a new log
I20260928 15:51:55.499877 705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0e640aac644f24bc0d63de1cb3d0d5" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43325 } }
I20260928 15:51:55.500298 705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:55.500415 705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0e640aac644f24bc0d63de1cb3d0d5, State: Initialized, Role: FOLLOWER
I20260928 15:51:55.501036 705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0e640aac644f24bc0d63de1cb3d0d5" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43325 } }
I20260928 15:51:55.501278 705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:55.501420 705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:55.501601 705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:55.503790 705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0e640aac644f24bc0d63de1cb3d0d5" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43325 } }
I20260928 15:51:55.504362 705 leader_election.cc:302] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e0e640aac644f24bc0d63de1cb3d0d5; no voters:
I20260928 15:51:55.505019 705 leader_election.cc:288] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:55.505224 711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:55.506050 711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3e0e640aac644f24bc0d63de1cb3d0d5, State: Running, Role: LEADER
I20260928 15:51:55.506610 711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0e640aac644f24bc0d63de1cb3d0d5" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43325 } }
I20260928 15:51:55.507288 705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:55.509843 713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e0e640aac644f24bc0d63de1cb3d0d5. Latest consensus state: current_term: 1 leader_uuid: "3e0e640aac644f24bc0d63de1cb3d0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0e640aac644f24bc0d63de1cb3d0d5" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43325 } } }
I20260928 15:51:55.510208 713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:55.510951 712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e0e640aac644f24bc0d63de1cb3d0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0e640aac644f24bc0d63de1cb3d0d5" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43325 } } }
I20260928 15:51:55.511348 712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:55.512550 718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:55.518388 718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:55.526820 718 catalog_manager.cc:1347] Generated new cluster ID: 90e3a8baa035451fa42f4b70aef6e8ff
I20260928 15:51:55.526965 718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:55.546265 718 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:55.546747 718 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:55.560201 718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e0e640aac644f24bc0d63de1cb3d0d5: Generated new TSK 0
I20260928 15:51:55.561249 718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:51:55.574827 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43325
--builtin_ntp_servers=127.22.157.212:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 15:51:55.927258 731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:55.927560 731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:55.927713 731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:55.935246 731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:55.935350 731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:55.935526 731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:55.944974 731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:55.946314 731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:55.948118 731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:55.961665 736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:55.961726 739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:55.962584 731 server_base.cc:1034] running on GCE node
W20260928 15:51:55.963260 737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:55.964001 731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:55.965082 731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:55.966312 731 hybrid_clock.cc:648] HybridClock initialized: now 1790610715966289 us; error 51 us; skew 500 ppm
I20260928 15:51:55.966660 731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:55.969090 731 webserver.cc:460] Webserver started at http://127.22.157.193:46583/ using document root <none> and password file <none>
I20260928 15:51:55.969848 731 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:55.969987 731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:55.970316 731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:55.975229 731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "50da6974d6c444df9356294d585f39e5"
format_stamp: "Formatted at 2026-09-28 15:51:55 on dist-test-slave-1vw0"
server_key: "52ca34c2a7372c4ffd5de269f25fcc4c"
server_key_iv: "2f07e9cb9541b70fd5d076757b76700c"
server_key_version: "encryptionkey@0"
I20260928 15:51:55.976049 731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "50da6974d6c444df9356294d585f39e5"
format_stamp: "Formatted at 2026-09-28 15:51:55 on dist-test-slave-1vw0"
server_key: "52ca34c2a7372c4ffd5de269f25fcc4c"
server_key_iv: "2f07e9cb9541b70fd5d076757b76700c"
server_key_version: "encryptionkey@0"
I20260928 15:51:55.981343 731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:51:55.984941 745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:55.986332 731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:55.986522 731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "50da6974d6c444df9356294d585f39e5"
format_stamp: "Formatted at 2026-09-28 15:51:55 on dist-test-slave-1vw0"
server_key: "52ca34c2a7372c4ffd5de269f25fcc4c"
server_key_iv: "2f07e9cb9541b70fd5d076757b76700c"
server_key_version: "encryptionkey@0"
I20260928 15:51:55.986773 731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:55.992621 731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:55.993463 731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:55.993741 731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:55.994671 731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:55.996083 731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:55.996200 731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:55.996335 731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:55.996420 731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:56.031625 731 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:36921
I20260928 15:51:56.031647 858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:36921 every 8 connection(s)
I20260928 15:51:56.033051 731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:56.034682 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 731
I20260928 15:51:56.034888 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:51:56.035727 23159 external_mini_cluster.cc:1428] Setting key 78e01ee88d1d0665d777c843d875e666
I20260928 15:51:56.038916 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43325
--builtin_ntp_servers=127.22.157.212:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 15:51:56.042892 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:56.036054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:53585 (local address 127.22.157.254:43325)
0928 15:51:56.036627 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:56.036633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:56.038149 (+ 1516us) server_negotiation.cc:407] Connection header received
0928 15:51:56.040185 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:56.040191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:56.040264 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:56.040406 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:56.041964 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:56.041971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:56.041976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:56.041998 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:56.042057 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:56.042070 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:56.042221 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:56.042445 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 15:51:56.042535 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":287,"threads_started":1}
I20260928 15:51:56.044005 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:56.036709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43325 (local address 127.22.157.193:53585)
0928 15:51:56.037943 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:56.038034 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:56.039874 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:56.040679 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:56.040689 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:56.040859 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:56.041634 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:56.041637 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:56.041680 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:56.041776 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:56.042178 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:56.042190 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:56.042200 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:51:56.042550 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 15:51:56.042877 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":304,"threads_started":1}
I20260928 15:51:56.045699 859 heartbeater.cc:347] Connected to a master server at 127.22.157.254:43325
I20260928 15:51:56.046023 859 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:56.046653 859 heartbeater.cc:511] Master 127.22.157.254:43325 requested a full tablet report, sending...
I20260928 15:51:56.048993 670 ts_manager.cc:194] Registered new tserver with Master: 50da6974d6c444df9356294d585f39e5 (127.22.157.193:36921)
W20260928 15:51:56.389240 863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:56.389542 863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:56.389643 863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:56.397325 863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:56.397433 863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:56.397625 863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:51:56.407001 863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:56.408432 863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:56.410338 863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:56.424336 868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:56.424407 871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:56.424386 869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:56.425722 863 server_base.cc:1034] running on GCE node
I20260928 15:51:56.426349 863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:56.427322 863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:56.428550 863 hybrid_clock.cc:648] HybridClock initialized: now 1790610716428511 us; error 48 us; skew 500 ppm
I20260928 15:51:56.428948 863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:56.431324 863 webserver.cc:460] Webserver started at http://127.22.157.194:32887/ using document root <none> and password file <none>
I20260928 15:51:56.432055 863 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:56.432189 863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:56.432518 863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:56.435612 863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "32b50254f23e4ec99e2c29ed0e692621"
format_stamp: "Formatted at 2026-09-28 15:51:56 on dist-test-slave-1vw0"
server_key: "f65d16dca91b8f0c3d7de8004d55ccc4"
server_key_iv: "c1c3ea512b4c066bfa7c0a53c818b249"
server_key_version: "encryptionkey@0"
I20260928 15:51:56.436442 863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "32b50254f23e4ec99e2c29ed0e692621"
format_stamp: "Formatted at 2026-09-28 15:51:56 on dist-test-slave-1vw0"
server_key: "f65d16dca91b8f0c3d7de8004d55ccc4"
server_key_iv: "c1c3ea512b4c066bfa7c0a53c818b249"
server_key_version: "encryptionkey@0"
I20260928 15:51:56.441774 863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 15:51:56.445420 877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:56.446980 863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:51:56.447199 863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "32b50254f23e4ec99e2c29ed0e692621"
format_stamp: "Formatted at 2026-09-28 15:51:56 on dist-test-slave-1vw0"
server_key: "f65d16dca91b8f0c3d7de8004d55ccc4"
server_key_iv: "c1c3ea512b4c066bfa7c0a53c818b249"
server_key_version: "encryptionkey@0"
I20260928 15:51:56.447501 863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:56.453529 863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:56.454366 863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:56.454663 863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:56.455649 863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:56.457166 863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:56.457284 863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:56.457422 863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:56.457509 863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:56.493628 863 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:43303
I20260928 15:51:56.493716 989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:43303 every 8 connection(s)
I20260928 15:51:56.494952 863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:51:56.496438 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 863
I20260928 15:51:56.496655 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:51:56.497296 23159 external_mini_cluster.cc:1428] Setting key dc773cf68331a5261757c22a677fe6ee
I20260928 15:51:56.500680 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43325
--builtin_ntp_servers=127.22.157.212:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 15:51:56.504989 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:56.498036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:44961 (local address 127.22.157.254:43325)
0928 15:51:56.498235 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:56.498241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:56.499926 (+ 1685us) server_negotiation.cc:407] Connection header received
0928 15:51:56.502666 (+ 2740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:56.502671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:56.502724 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:56.502809 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:56.504261 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:56.504266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:56.504269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:56.504286 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:56.504318 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:56.504329 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:56.504424 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:56.504654 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 15:51:56.504703 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:51:56.506270 993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:56.498673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43325 (local address 127.22.157.194:44961)
0928 15:51:56.499747 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:56.499803 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:56.502364 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:56.503054 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:56.503069 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:56.503205 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:56.503896 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:56.503901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:56.503952 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:56.504066 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:56.504429 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:56.504442 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:56.504452 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:51:56.504763 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 15:51:56.505028 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":292,"threads_started":1}
I20260928 15:51:56.507905 991 heartbeater.cc:347] Connected to a master server at 127.22.157.254:43325
I20260928 15:51:56.508232 991 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:56.508965 991 heartbeater.cc:511] Master 127.22.157.254:43325 requested a full tablet report, sending...
I20260928 15:51:56.510346 670 ts_manager.cc:194] Registered new tserver with Master: 32b50254f23e4ec99e2c29ed0e692621 (127.22.157.194:43303)
W20260928 15:51:56.850387 994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:56.850757 994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:56.850929 994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:56.858556 994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:56.858716 994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:56.858902 994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:51:56.868484 994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:56.869905 994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:56.871845 994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:56.885769 999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:56.885836 1000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:56.885798 1002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:56.887061 994 server_base.cc:1034] running on GCE node
I20260928 15:51:56.887712 994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:56.888798 994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:56.890048 994 hybrid_clock.cc:648] HybridClock initialized: now 1790610716889986 us; error 53 us; skew 500 ppm
I20260928 15:51:56.890489 994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:51:56.893198 994 webserver.cc:460] Webserver started at http://127.22.157.195:44975/ using document root <none> and password file <none>
I20260928 15:51:56.894097 994 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:56.894243 994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:56.894608 994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:56.897706 994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "a17cbc54b5fb478eb383ea04054e01fd"
format_stamp: "Formatted at 2026-09-28 15:51:56 on dist-test-slave-1vw0"
server_key: "529f372e36b4267b4cd4bef58699aa62"
server_key_iv: "53433828148953171f3137f535aa9e4d"
server_key_version: "encryptionkey@0"
I20260928 15:51:56.898543 994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "a17cbc54b5fb478eb383ea04054e01fd"
format_stamp: "Formatted at 2026-09-28 15:51:56 on dist-test-slave-1vw0"
server_key: "529f372e36b4267b4cd4bef58699aa62"
server_key_iv: "53433828148953171f3137f535aa9e4d"
server_key_version: "encryptionkey@0"
I20260928 15:51:56.903995 994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:51:56.907601 1008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:56.909149 994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:56.909314 994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "a17cbc54b5fb478eb383ea04054e01fd"
format_stamp: "Formatted at 2026-09-28 15:51:56 on dist-test-slave-1vw0"
server_key: "529f372e36b4267b4cd4bef58699aa62"
server_key_iv: "53433828148953171f3137f535aa9e4d"
server_key_version: "encryptionkey@0"
I20260928 15:51:56.909533 994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:56.915369 994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:56.916172 994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:56.916447 994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:56.917521 994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:56.918970 994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:56.919062 994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:56.919138 994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:56.919194 994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:56.952625 994 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:42537
I20260928 15:51:56.952716 1120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:42537 every 8 connection(s)
I20260928 15:51:56.954047 994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:51:56.958775 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 994
I20260928 15:51:56.958962 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:51:56.959487 23159 external_mini_cluster.cc:1428] Setting key 78b51d041c9e0c5166fe94dfacb38048
I20260928 15:51:56.962949 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:56.956625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:49249 (local address 127.22.157.254:43325)
0928 15:51:56.956877 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:56.956883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:56.958325 (+ 1442us) server_negotiation.cc:407] Connection header received
0928 15:51:56.960597 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:56.960603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:56.960679 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:56.960822 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:56.962212 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:56.962216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:56.962219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:56.962236 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:56.962267 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:56.962279 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:56.962398 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:56.962610 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 15:51:56.962678 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:51:56.963871 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:56.957237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43325 (local address 127.22.157.195:49249)
0928 15:51:56.958137 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:56.958191 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:56.960312 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:56.961048 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:56.961057 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:56.961160 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:56.961849 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:51:56.961853 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:56.961905 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:51:56.962015 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:56.962383 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:56.962396 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:56.962406 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:51:56.962687 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 15:51:56.962946 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":253,"threads_started":1}
I20260928 15:51:56.965142 1121 heartbeater.cc:347] Connected to a master server at 127.22.157.254:43325
I20260928 15:51:56.965440 1121 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:56.966069 1121 heartbeater.cc:511] Master 127.22.157.254:43325 requested a full tablet report, sending...
I20260928 15:51:56.967402 670 ts_manager.cc:194] Registered new tserver with Master: a17cbc54b5fb478eb383ea04054e01fd (127.22.157.195:42537)
I20260928 15:51:56.974999 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 15:51:56.981040 861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:51:56.979458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.22.157.254:43325)
0928 15:51:56.979692 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:56.979701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:56.980136 (+ 435us) server_negotiation.cc:407] Connection header received
0928 15:51:56.980265 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:56.980270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:56.980321 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:56.980398 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:56.980797 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42302: BlockingRecv error: recv got EOF from 127.0.0.1:42302 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:51:56.981834 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 731
I20260928 15:51:56.985502 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:56.982833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.22.157.193:36921)
0928 15:51:56.983333 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:56.983353 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:56.983377 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:51:56.983433 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:56.983439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:56.983667 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:56.983788 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:56.984681 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:56.984688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:56.984699 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:56.984718 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:56.984820 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:56.984845 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:56.984988 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:56.985098 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 15:51:56.985209 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
I20260928 15:51:56.986665 854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:57.068418 859 heartbeater.cc:503] Master 127.22.157.254:43325 was elected leader, sending a full tablet report...
I20260928 15:51:57.073232 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 731
I20260928 15:51:57.090112 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 863
I20260928 15:51:57.094218 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:57.091310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.22.157.194:43303)
0928 15:51:57.091908 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:57.091915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:57.091941 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:51:57.092004 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:57.092009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:57.092255 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:57.092381 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:57.093429 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:57.093435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:57.093447 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:57.093467 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:57.093512 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:57.093526 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:57.093641 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:57.093735 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 15:51:57.093855 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":326,"threads_started":1}
I20260928 15:51:57.095444 985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:57.178982 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 863
I20260928 15:51:57.195201 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 994
I20260928 15:51:57.199314 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:57.196254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.22.157.195:42537)
0928 15:51:57.197013 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:57.197021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:57.197054 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:51:57.197117 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:57.197123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:57.197345 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:57.197472 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:57.198549 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:57.198556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:57.198572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:51:57.198590 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:57.198631 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:51:57.198644 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:57.198778 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:57.198841 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 15:51:57.198963 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":377,"threads_started":1}
I20260928 15:51:57.200634 1116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:57.284268 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 994
I20260928 15:51:57.302304 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 639
I20260928 15:51:57.303758 700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:51:57.408937 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 639
2026-09-28T15:51:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2459 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:51:57 dist-test-slave-1vw0 krb5kdc[1140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:51:57 dist-test-slave-1vw0 krb5kdc[1140](info): set up 2 sockets
Sep 28 15:51:57 dist-test-slave-1vw0 krb5kdc[1140](info): commencing operation
krb5kdc: starting...
W20260928 15:51:58.542189 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:51:58 dist-test-slave-1vw0 krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:51:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:51:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:51:58.809516 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:35795
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:35795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:59.154254 1153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:59.154561 1153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:59.154651 1153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:59.161839 1153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:51:59.161962 1153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:59.162009 1153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:59.162060 1153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:51:59.162104 1153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:51:59.172211 1153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:35795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:35795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:59.173542 1153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:59.174903 1153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:59.187001 1159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:59.187057 1161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:59.187080 1158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:59.187935 1153 server_base.cc:1034] running on GCE node
I20260928 15:51:59.188822 1153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:59.190304 1153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:59.191553 1153 hybrid_clock.cc:648] HybridClock initialized: now 1790610719191505 us; error 52 us; skew 500 ppm
I20260928 15:51:59.191912 1153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:59 dist-test-slave-1vw0 krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610719, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:59.207864 1153 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:51:59.209625 1153 webserver.cc:460] Webserver started at http://127.22.157.254:37187/ using document root <none> and password file <none>
I20260928 15:51:59.210412 1153 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:59.210578 1153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:59.210919 1153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:59.214130 1153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "97c8c68bc269468c914b40c9371d0fa0"
format_stamp: "Formatted at 2026-09-28 15:51:59 on dist-test-slave-1vw0"
server_key: "11501f098bc46e5c32b86ba48b1d5f62"
server_key_iv: "da16b14882f822a5505e520b331de794"
server_key_version: "encryptionkey@0"
I20260928 15:51:59.214958 1153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "97c8c68bc269468c914b40c9371d0fa0"
format_stamp: "Formatted at 2026-09-28 15:51:59 on dist-test-slave-1vw0"
server_key: "11501f098bc46e5c32b86ba48b1d5f62"
server_key_iv: "da16b14882f822a5505e520b331de794"
server_key_version: "encryptionkey@0"
I20260928 15:51:59.220515 1153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:51:59.224351 1168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:59.225927 1153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:59.226141 1153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "97c8c68bc269468c914b40c9371d0fa0"
format_stamp: "Formatted at 2026-09-28 15:51:59 on dist-test-slave-1vw0"
server_key: "11501f098bc46e5c32b86ba48b1d5f62"
server_key_iv: "da16b14882f822a5505e520b331de794"
server_key_version: "encryptionkey@0"
I20260928 15:51:59.226351 1153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:59.245326 1153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:59.249522 1153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:59.249825 1153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:59.268607 1153 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:35795
I20260928 15:51:59.268652 1219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:35795 every 8 connection(s)
I20260928 15:51:59.270112 1153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:51:59.273916 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:51:59.276037 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 1153
I20260928 15:51:59.276268 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:51:59.276952 23159 external_mini_cluster.cc:1428] Setting key 3b7a3523a1ee44761892418ea1377548
I20260928 15:51:59.285979 1220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0: Bootstrap starting.
I20260928 15:51:59.289948 1220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0: Neither blocks nor log segments found. Creating new log.
I20260928 15:51:59.291354 1220 log.cc:824] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:51:59 dist-test-slave-1vw0 krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:51:59.294486 1220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0: No bootstrap required, opened a new log
I20260928 15:51:59.301105 1223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:59.279005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.22.157.254:35795)
0928 15:51:59.279853 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:59.279864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:59.279908 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:51:59.281618 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:59.281640 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:59.282035 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:59.282365 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:59.284252 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:59.285760 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:59.287545 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:59.288270 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:59.288403 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:59.293484 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:59.293492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:59.293508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:59.293556 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:59.296830 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:59.297556 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:59.297570 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:59.297583 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:59.297754 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:59.298142 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:59.298147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:59.298152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:59.298735 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:59.299081 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:59.299521 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 15:51:59.299962 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":383,"threads_started":1}
I20260928 15:51:59.302729 1220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c68bc269468c914b40c9371d0fa0" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35795 } }
I20260928 15:51:59.303118 1220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:51:59.303242 1220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c8c68bc269468c914b40c9371d0fa0, State: Initialized, Role: FOLLOWER
I20260928 15:51:59.303884 1220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c68bc269468c914b40c9371d0fa0" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35795 } }
I20260928 15:51:59.304162 1220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:51:59.304299 1220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:51:59.304427 1220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:51:59.307004 1220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c68bc269468c914b40c9371d0fa0" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35795 } }
I20260928 15:51:59.307502 1220 leader_election.cc:302] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97c8c68bc269468c914b40c9371d0fa0; no voters:
I20260928 15:51:59.308035 1220 leader_election.cc:288] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:51:59.308192 1225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:51:59.309425 1225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [term 1 LEADER]: Becoming Leader. State: Replica: 97c8c68bc269468c914b40c9371d0fa0, State: Running, Role: LEADER
I20260928 15:51:59.310002 1225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c68bc269468c914b40c9371d0fa0" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35795 } }
I20260928 15:51:59.310492 1220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:51:59.313195 1227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97c8c68bc269468c914b40c9371d0fa0. Latest consensus state: current_term: 1 leader_uuid: "97c8c68bc269468c914b40c9371d0fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c68bc269468c914b40c9371d0fa0" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35795 } } }
I20260928 15:51:59.313187 1226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97c8c68bc269468c914b40c9371d0fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c8c68bc269468c914b40c9371d0fa0" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35795 } } }
I20260928 15:51:59.313580 1227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:59.313643 1226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0 [sys.catalog]: This master's current role is: LEADER
I20260928 15:51:59.316217 1231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:51:59.321904 1231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:51:59.330250 1231 catalog_manager.cc:1347] Generated new cluster ID: 2d47444007144b5780c9387f70d5cc5b
I20260928 15:51:59.330394 1231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:51:59.345017 1231 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:51:59.346580 1231 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:51:59.363947 1231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0: Generated new TSK 0
I20260928 15:51:59.365531 1231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:51:59.449292 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35795
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:51:59.830160 1246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:51:59.830472 1246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:51:59.830636 1246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:51:59.838356 1246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:51:59.838488 1246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:51:59.838661 1246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:51:59.848932 1246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:51:59.850354 1246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:51:59.851792 1246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:51:59.867929 1252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:59.867985 1251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:51:59.868077 1254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:51:59.869030 1246 server_base.cc:1034] running on GCE node
I20260928 15:51:59.869656 1246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:51:59.870733 1246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:51:59.871989 1246 hybrid_clock.cc:648] HybridClock initialized: now 1790610719871936 us; error 56 us; skew 500 ppm
I20260928 15:51:59.872365 1246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:51:59 dist-test-slave-1vw0 krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610719, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:51:59.875844 1246 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:51:59.877485 1246 webserver.cc:460] Webserver started at http://127.22.157.193:33111/ using document root <none> and password file <none>
I20260928 15:51:59.878186 1246 fs_manager.cc:359] Metadata directory not provided
I20260928 15:51:59.878291 1246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:51:59.878561 1246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:51:59.881682 1246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "9734394e456347eb957c4c742fb3646e"
format_stamp: "Formatted at 2026-09-28 15:51:59 on dist-test-slave-1vw0"
server_key: "7b054687ffa890f52156185dbe25c47a"
server_key_iv: "af9a3e942e96119ad0ece019e6beb73c"
server_key_version: "encryptionkey@0"
I20260928 15:51:59.882457 1246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "9734394e456347eb957c4c742fb3646e"
format_stamp: "Formatted at 2026-09-28 15:51:59 on dist-test-slave-1vw0"
server_key: "7b054687ffa890f52156185dbe25c47a"
server_key_iv: "af9a3e942e96119ad0ece019e6beb73c"
server_key_version: "encryptionkey@0"
I20260928 15:51:59.887825 1246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:51:59.891464 1261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:59.892985 1246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:51:59.893141 1246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "9734394e456347eb957c4c742fb3646e"
format_stamp: "Formatted at 2026-09-28 15:51:59 on dist-test-slave-1vw0"
server_key: "7b054687ffa890f52156185dbe25c47a"
server_key_iv: "af9a3e942e96119ad0ece019e6beb73c"
server_key_version: "encryptionkey@0"
I20260928 15:51:59.893313 1246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:51:59.906915 1246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:51:59.911492 1246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:51:59.911818 1246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:51:59.912772 1246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:51:59.914090 1246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:51:59.914178 1246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:59.914266 1246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:51:59.914314 1246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:51:59.949199 1246 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:44967
I20260928 15:51:59.949229 1375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:44967 every 8 connection(s)
I20260928 15:51:59.950645 1246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:51:59.956588 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 1246
I20260928 15:51:59.956846 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:51:59.957504 23159 external_mini_cluster.cc:1428] Setting key 512f6cadd582badf0b7c3277940fee50
Sep 28 15:51:59 dist-test-slave-1vw0 krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610719, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:51:59.973464 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:59.953457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:56335 (local address 127.22.157.254:35795)
0928 15:51:59.953934 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 15:51:59.953939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:51:59.954802 (+ 863us) server_negotiation.cc:407] Connection header received
0928 15:51:59.956624 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:51:59.956630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:51:59.956785 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:51:59.956919 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:51:59.958690 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:59.959529 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:59.961393 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:51:59.962137 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:51:59.962280 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:59.966300 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:51:59.966309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:51:59.966317 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:51:59.966366 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:51:59.969706 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:59.970785 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:59.970789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:59.970792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:59.970929 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:51:59.971458 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:51:59.971462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:51:59.971464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:51:59.971792 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:51:59.971921 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 15:51:59.972973 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0928 15:51:59.973173 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":253,"threads_started":1}
I20260928 15:51:59.975186 1379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:51:59.953897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35795 (local address 127.22.157.193:56335)
0928 15:51:59.954640 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 15:51:59.954678 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:51:59.956344 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:51:59.957180 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:51:59.957189 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:51:59.957791 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:51:59.958511 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:59.958533 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:59.959713 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:59.959718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:59.961224 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:51:59.961238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:51:59.962380 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:51:59.962388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:51:59.962631 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:51:59.963417 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:51:59.963446 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:51:59.966057 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:51:59.969893 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:59.969907 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:59.969930 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:59.970633 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:59.971050 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:51:59.971057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:51:59.971064 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:51:59.971341 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:51:59.971924 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:51:59.971934 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:51:59.972478 (+ 544us) client_negotiation.cc:769] Sending connection context
0928 15:51:59.972872 (+ 394us) client_negotiation.cc:240] Negotiation successful
0928 15:51:59.973502 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":275,"threads_started":1}
I20260928 15:51:59.977087 1376 heartbeater.cc:347] Connected to a master server at 127.22.157.254:35795
I20260928 15:51:59.977442 1376 heartbeater.cc:464] Registering TS with master...
I20260928 15:51:59.978282 1376 heartbeater.cc:511] Master 127.22.157.254:35795 requested a full tablet report, sending...
I20260928 15:51:59.980700 1185 ts_manager.cc:194] Registered new tserver with Master: 9734394e456347eb957c4c742fb3646e (127.22.157.193:44967)
I20260928 15:51:59.982563 1185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:56335
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:52:00.021301 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35795
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:52:00.362267 1382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:52:00.362587 1382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:52:00.362692 1382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:52:00.370361 1382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:52:00.370487 1382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:52:00.370612 1382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:52:00.380896 1382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:52:00.382200 1382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:52:00.383491 1382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:52:00.397399 1390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:00.397459 1388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:00.397471 1387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:52:00.398398 1382 server_base.cc:1034] running on GCE node
I20260928 15:52:00.399111 1382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:52:00.400141 1382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:52:00.401381 1382 hybrid_clock.cc:648] HybridClock initialized: now 1790610720401329 us; error 49 us; skew 500 ppm
I20260928 15:52:00.401865 1382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:52:00 dist-test-slave-1vw0 krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610720, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:52:00.405349 1382 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:52:00.407196 1382 webserver.cc:460] Webserver started at http://127.22.157.194:39041/ using document root <none> and password file <none>
I20260928 15:52:00.407972 1382 fs_manager.cc:359] Metadata directory not provided
I20260928 15:52:00.408133 1382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:52:00.408457 1382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:52:00.411734 1382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "b507c27af3ad4da499539f0621f698f5"
format_stamp: "Formatted at 2026-09-28 15:52:00 on dist-test-slave-1vw0"
server_key: "8e97fd46baabb545e6e38cf9cde0d290"
server_key_iv: "d94987ec6a7e09c46ba86fb905738bf2"
server_key_version: "encryptionkey@0"
I20260928 15:52:00.412568 1382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "b507c27af3ad4da499539f0621f698f5"
format_stamp: "Formatted at 2026-09-28 15:52:00 on dist-test-slave-1vw0"
server_key: "8e97fd46baabb545e6e38cf9cde0d290"
server_key_iv: "d94987ec6a7e09c46ba86fb905738bf2"
server_key_version: "encryptionkey@0"
I20260928 15:52:00.418902 1382 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 15:52:00.423167 1397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:00.424881 1382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:52:00.425077 1382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "b507c27af3ad4da499539f0621f698f5"
format_stamp: "Formatted at 2026-09-28 15:52:00 on dist-test-slave-1vw0"
server_key: "8e97fd46baabb545e6e38cf9cde0d290"
server_key_iv: "d94987ec6a7e09c46ba86fb905738bf2"
server_key_version: "encryptionkey@0"
I20260928 15:52:00.425308 1382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:52:00.456549 1382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:52:00.461283 1382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:52:00.461548 1382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:52:00.462509 1382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:52:00.463985 1382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:52:00.464061 1382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:00.464154 1382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:52:00.464196 1382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:00.498708 1382 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:35229
I20260928 15:52:00.498760 1511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:35229 every 8 connection(s)
I20260928 15:52:00.500013 1382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:52:00.507978 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 1382
I20260928 15:52:00.508229 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:52:00.508790 23159 external_mini_cluster.cc:1428] Setting key a4bdd76c90819f6fccc9a6d3e7caf8ba
Sep 28 15:52:00 dist-test-slave-1vw0 krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610720, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:52:00.522189 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:00.502792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:55349 (local address 127.22.157.254:35795)
0928 15:52:00.503250 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:00.503257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:00.504326 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 15:52:00.506443 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:00.506449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:00.506551 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:00.506682 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:00.508453 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:00.509303 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:00.511417 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:00.511919 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:00.512107 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:00.515875 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:00.515881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:00.515884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:00.515918 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:00.518984 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:00.519778 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:00.519785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:00.519789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:00.519926 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:00.520364 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:00.520370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:00.520373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:00.520658 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:00.520836 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:00.521569 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 15:52:00.521812 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":209,"threads_started":1}
I20260928 15:52:00.523352 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:00.503416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35795 (local address 127.22.157.194:55349)
0928 15:52:00.504149 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:00.504193 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:00.506138 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:00.506952 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:00.506961 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:00.507518 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:52:00.508234 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:00.508255 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:00.509460 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:00.509466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:00.511266 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:00.511280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:00.512209 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:00.512218 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:00.512509 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:00.513335 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:52:00.513359 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:52:00.515638 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:52:00.519172 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:00.519181 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:00.519201 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:00.519648 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:00.520040 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:00.520045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:00.520049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:00.520247 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:00.520806 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:52:00.520815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:52:00.521200 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 15:52:00.521483 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 15:52:00.522067 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":282,"threads_started":1}
I20260928 15:52:00.525091 1512 heartbeater.cc:347] Connected to a master server at 127.22.157.254:35795
I20260928 15:52:00.525415 1512 heartbeater.cc:464] Registering TS with master...
I20260928 15:52:00.526182 1512 heartbeater.cc:511] Master 127.22.157.254:35795 requested a full tablet report, sending...
I20260928 15:52:00.527806 1185 ts_manager.cc:194] Registered new tserver with Master: b507c27af3ad4da499539f0621f698f5 (127.22.157.194:35229)
I20260928 15:52:00.529261 1185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:55349
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:52:00.572860 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35795
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:52:00.917363 1518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:52:00.917732 1518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:52:00.917833 1518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:52:00.925580 1518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:52:00.925707 1518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:52:00.925824 1518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:52:00.936466 1518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:52:00.937953 1518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:52:00.939285 1518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:52:00.953743 1524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:00.953722 1526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:52:00.955055 1518 server_base.cc:1034] running on GCE node
W20260928 15:52:00.955884 1523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:52:00.956657 1518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:52:00.957676 1518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:52:00.958923 1518 hybrid_clock.cc:648] HybridClock initialized: now 1790610720958845 us; error 70 us; skew 500 ppm
I20260928 15:52:00.959261 1518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:52:00 dist-test-slave-1vw0 krb5kdc[1140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610720, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:52:00.962772 1518 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:52:00.964378 1518 webserver.cc:460] Webserver started at http://127.22.157.195:38015/ using document root <none> and password file <none>
I20260928 15:52:00.965205 1518 fs_manager.cc:359] Metadata directory not provided
I20260928 15:52:00.965353 1518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:52:00.965669 1518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:52:00.968818 1518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "64fd823a7656431898d2f85305fa81cb"
format_stamp: "Formatted at 2026-09-28 15:52:00 on dist-test-slave-1vw0"
server_key: "19640f378a873d79402154ad800fa26f"
server_key_iv: "559e41a56f94778ec77c591d2e831b63"
server_key_version: "encryptionkey@0"
I20260928 15:52:00.969666 1518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "64fd823a7656431898d2f85305fa81cb"
format_stamp: "Formatted at 2026-09-28 15:52:00 on dist-test-slave-1vw0"
server_key: "19640f378a873d79402154ad800fa26f"
server_key_iv: "559e41a56f94778ec77c591d2e831b63"
server_key_version: "encryptionkey@0"
I20260928 15:52:00.975029 1518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 15:52:00.978796 1533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:00.980333 1518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:52:00.980535 1518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "64fd823a7656431898d2f85305fa81cb"
format_stamp: "Formatted at 2026-09-28 15:52:00 on dist-test-slave-1vw0"
server_key: "19640f378a873d79402154ad800fa26f"
server_key_iv: "559e41a56f94778ec77c591d2e831b63"
server_key_version: "encryptionkey@0"
I20260928 15:52:00.981005 1518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:52:00.986068 1376 heartbeater.cc:503] Master 127.22.157.254:35795 was elected leader, sending a full tablet report...
I20260928 15:52:01.014672 1518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:52:01.019541 1518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:52:01.019806 1518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:52:01.020836 1518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:52:01.022219 1518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:52:01.022296 1518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:01.022383 1518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:52:01.022437 1518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:01.058597 1518 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:34583
I20260928 15:52:01.058648 1646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:34583 every 8 connection(s)
I20260928 15:52:01.059967 1518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:52:01.060381 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 1518
I20260928 15:52:01.060596 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:52:01.061256 23159 external_mini_cluster.cc:1428] Setting key 334e251da0ad17536a0b7e87aa258845
Sep 28 15:52:01 dist-test-slave-1vw0 krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610720, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:52:01.082350 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.062811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:59545 (local address 127.22.157.254:35795)
0928 15:52:01.063350 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.063356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.064794 (+ 1438us) server_negotiation.cc:407] Connection header received
0928 15:52:01.066515 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.066519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.066606 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.066702 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:01.068200 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.069080 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.070770 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.071173 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.071268 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.075963 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:01.075971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:01.075978 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:01.076026 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:01.079178 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.079920 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.079927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.079931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.080065 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.080510 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.080517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.080520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.080876 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:01.081033 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.081732 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.081952 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":264,"threads_started":1}
I20260928 15:52:01.083334 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.063551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35795 (local address 127.22.157.195:59545)
0928 15:52:01.064410 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.064469 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:01.066218 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:01.066919 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:01.066927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:01.067461 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:52:01.068041 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.068061 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.069246 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.069250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.070633 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.070648 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.071443 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.071450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.071707 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.072758 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:52:01.072790 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:52:01.075729 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:52:01.079342 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:01.079351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:01.079369 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:01.079799 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:01.080192 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:01.080197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:01.080200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:01.080398 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:01.081001 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:52:01.081009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:52:01.081365 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 15:52:01.081650 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 15:52:01.082198 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":307,"threads_started":1}
I20260928 15:52:01.084843 1647 heartbeater.cc:347] Connected to a master server at 127.22.157.254:35795
I20260928 15:52:01.085153 1647 heartbeater.cc:464] Registering TS with master...
I20260928 15:52:01.085966 1647 heartbeater.cc:511] Master 127.22.157.254:35795 requested a full tablet report, sending...
I20260928 15:52:01.087474 1185 ts_manager.cc:194] Registered new tserver with Master: 64fd823a7656431898d2f85305fa81cb (127.22.157.195:34583)
I20260928 15:52:01.088658 1185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:59545
I20260928 15:52:01.091846 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:52:01.109256 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.096664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.22.157.254:35795)
0928 15:52:01.096946 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.096953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.097228 (+ 275us) server_negotiation.cc:407] Connection header received
0928 15:52:01.097387 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.097391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.097455 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.097535 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:01.099153 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.099894 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.101164 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.101544 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.101635 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.104051 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:01.104056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:01.104060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:01.104098 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:01.106719 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.107375 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.107380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.107382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.107509 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.107829 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.107832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.107834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.108142 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:01.108322 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.108653 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.108891 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 15:52:01.113698 1185 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:52:01.121104 1185 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:52:01.150312 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.134976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.22.157.195:34583)
0928 15:52:01.141061 (+ 6085us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.141069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.141097 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:52:01.141206 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.141215 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.141505 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.141793 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:01.142734 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.144658 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.147073 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.148945 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.149296 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.149468 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.149753 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.149921 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5924,"thread_start_us":5835,"threads_started":1}
I20260928 15:52:01.150350 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.134484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:34583 (local address 127.0.0.1:46768)
0928 15:52:01.135667 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.135716 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:01.135883 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:01.141814 (+ 5931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:01.141822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:01.141876 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:01.142487 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.142515 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.144822 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.144828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.146822 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.146867 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.149245 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.149256 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.149471 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.149535 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 15:52:01.149699 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 15:52:01.149921 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":71,"spinlock_wait_cycles":9472,"thread_start_us":803,"threads_started":1}
I20260928 15:52:01.151346 1664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.134953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.0.0.1:33488)
0928 15:52:01.136614 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.136637 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:01.136779 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:01.141815 (+ 5036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:01.141822 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:01.141878 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:01.142491 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.142503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.144535 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.144540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.147324 (+ 2784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.147336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.150524 (+ 3188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.150533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.150671 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.150694 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:52:01.150807 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 15:52:01.150943 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"mutex_wait_us":142,"spinlock_wait_cycles":9728,"thread_start_us":6279,"threads_started":1}
I20260928 15:52:01.151392 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.135536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.22.157.193:44967)
0928 15:52:01.141018 (+ 5482us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.141030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.141067 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:52:01.141207 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.141215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.141505 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.141793 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:01.142756 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.144376 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.147503 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.148977 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.149181 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.149347 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.150887 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.151072 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5312,"thread_start_us":5120,"threads_started":1}
I20260928 15:52:01.152406 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.136074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:35229 (local address 127.0.0.1:37870)
0928 15:52:01.141482 (+ 5408us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.141502 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:01.141672 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:01.142771 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:01.142775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:01.142802 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:01.143189 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.143200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.145015 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.145023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.147186 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.147202 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.151593 (+ 4391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.151600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.151780 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.151818 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 15:52:01.151917 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 15:52:01.152055 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5277,"mutex_wait_us":3,"thread_start_us":703,"threads_started":1}
I20260928 15:52:01.153313 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.136613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.22.157.194:35229)
0928 15:52:01.140527 (+ 3914us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.140540 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.141632 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 15:52:01.142293 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.142304 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.142572 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.142776 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:01.143361 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.144865 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.147414 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.148974 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.149154 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.149347 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.152786 (+ 3439us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.152992 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3750,"thread_start_us":3623,"threads_started":1}
I20260928 15:52:01.155058 1581 tablet_service.cc:1467] Processing CreateTablet for tablet 428eaa12c6c94787a2bf72414329e0ee (DEFAULT_TABLE table=test-table [id=b90adf09597e49e89fecdd5d2390ffa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:52:01.155444 1309 tablet_service.cc:1467] Processing CreateTablet for tablet 428eaa12c6c94787a2bf72414329e0ee (DEFAULT_TABLE table=test-table [id=b90adf09597e49e89fecdd5d2390ffa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:52:01.156701 1581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428eaa12c6c94787a2bf72414329e0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:52:01.156785 1309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428eaa12c6c94787a2bf72414329e0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:52:01.156477 1446 tablet_service.cc:1467] Processing CreateTablet for tablet 428eaa12c6c94787a2bf72414329e0ee (DEFAULT_TABLE table=test-table [id=b90adf09597e49e89fecdd5d2390ffa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:52:01.157614 1446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 428eaa12c6c94787a2bf72414329e0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:52:01.169641 1666 tablet_bootstrap.cc:492] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: Bootstrap starting.
I20260928 15:52:01.172174 1668 tablet_bootstrap.cc:492] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: Bootstrap starting.
I20260928 15:52:01.173468 1667 tablet_bootstrap.cc:492] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e: Bootstrap starting.
I20260928 15:52:01.174166 1666 tablet_bootstrap.cc:654] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: Neither blocks nor log segments found. Creating new log.
I20260928 15:52:01.175637 1666 log.cc:824] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:52:01.177431 1667 tablet_bootstrap.cc:654] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e: Neither blocks nor log segments found. Creating new log.
I20260928 15:52:01.179059 1668 tablet_bootstrap.cc:654] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: Neither blocks nor log segments found. Creating new log.
I20260928 15:52:01.179049 1667 log.cc:824] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:52:01.180286 1668 log.cc:824] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:52:01.182765 1667 tablet_bootstrap.cc:492] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e: No bootstrap required, opened a new log
I20260928 15:52:01.183092 1667 ts_tablet_manager.cc:1397] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 15:52:01.183406 1666 tablet_bootstrap.cc:492] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: No bootstrap required, opened a new log
I20260928 15:52:01.183534 1668 tablet_bootstrap.cc:492] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: No bootstrap required, opened a new log
I20260928 15:52:01.183717 1666 ts_tablet_manager.cc:1397] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260928 15:52:01.183799 1668 ts_tablet_manager.cc:1397] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260928 15:52:01.191689 1666 raft_consensus.cc:348] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } }
I20260928 15:52:01.192138 1666 raft_consensus.cc:374] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:52:01.192263 1666 raft_consensus.cc:729] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b507c27af3ad4da499539f0621f698f5, State: Initialized, Role: FOLLOWER
I20260928 15:52:01.193018 1666 consensus_queue.cc:260] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } }
I20260928 15:52:01.194378 1512 heartbeater.cc:503] Master 127.22.157.254:35795 was elected leader, sending a full tablet report...
I20260928 15:52:01.194427 1667 raft_consensus.cc:348] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } }
I20260928 15:52:01.194938 1667 raft_consensus.cc:374] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:52:01.195051 1667 raft_consensus.cc:729] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9734394e456347eb957c4c742fb3646e, State: Initialized, Role: FOLLOWER
I20260928 15:52:01.195250 1666 ts_tablet_manager.cc:1428] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260928 15:52:01.195801 1667 consensus_queue.cc:260] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } }
I20260928 15:52:01.197041 1668 raft_consensus.cc:348] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } }
I20260928 15:52:01.197505 1668 raft_consensus.cc:374] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:52:01.197614 1668 raft_consensus.cc:729] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd823a7656431898d2f85305fa81cb, State: Initialized, Role: FOLLOWER
I20260928 15:52:01.198232 1667 ts_tablet_manager.cc:1428] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260928 15:52:01.198343 1668 consensus_queue.cc:260] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } }
I20260928 15:52:01.199501 1647 heartbeater.cc:503] Master 127.22.157.254:35795 was elected leader, sending a full tablet report...
W20260928 15:52:01.203330 1377 tablet.cc:2369] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:52:01.203728 1668 ts_tablet_manager.cc:1428] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: Time spent starting tablet: real 0.020s user 0.012s sys 0.004s
W20260928 15:52:01.252264 1513 tablet.cc:2369] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:52:01.276537 1672 raft_consensus.cc:482] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:52:01.276911 1672 raft_consensus.cc:504] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } }
I20260928 15:52:01.279197 1672 leader_election.cc:288] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64fd823a7656431898d2f85305fa81cb (127.22.157.195:34583), 9734394e456347eb957c4c742fb3646e (127.22.157.193:44967)
I20260928 15:52:01.288184 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.279686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:58989 (local address 127.22.157.193:44967)
0928 15:52:01.279894 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.279900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.280566 (+ 666us) server_negotiation.cc:407] Connection header received
0928 15:52:01.280950 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.280957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.281038 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.281131 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:01.281749 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.282849 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.285589 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.286943 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.287143 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.287224 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.287764 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.287912 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:52:01.288184 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.280008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:54139 (local address 127.22.157.195:34583)
0928 15:52:01.280240 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.280245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.280901 (+ 656us) server_negotiation.cc:407] Connection header received
0928 15:52:01.281173 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.281179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.281256 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.281351 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:01.281962 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.282686 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.285589 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.286988 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.287142 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.287224 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.287783 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.287915 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 15:52:01.288555 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.279652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.22.157.194:58989)
0928 15:52:01.280433 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.280457 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:01.280761 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:01.281173 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:01.281177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:01.281202 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:01.281580 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.281592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.282986 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.282991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.285385 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.285403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.287311 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.287318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.287520 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.287586 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 15:52:01.287696 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 15:52:01.287854 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":65,"thread_start_us":456,"threads_started":1}
I20260928 15:52:01.288576 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.279660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:34583 (local address 127.22.157.194:54139)
0928 15:52:01.280782 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.280804 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:01.281003 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:01.281370 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:01.281375 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:01.281394 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:01.281817 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.281826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.282824 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.282831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.285385 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:01.285403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.287292 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.287303 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:01.287518 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.287656 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 15:52:01.287736 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:52:01.287854 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":351,"thread_start_us":542,"threads_started":1}
I20260928 15:52:01.289619 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428eaa12c6c94787a2bf72414329e0ee" candidate_uuid: "b507c27af3ad4da499539f0621f698f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9734394e456347eb957c4c742fb3646e" is_pre_election: true
I20260928 15:52:01.289636 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428eaa12c6c94787a2bf72414329e0ee" candidate_uuid: "b507c27af3ad4da499539f0621f698f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd823a7656431898d2f85305fa81cb" is_pre_election: true
I20260928 15:52:01.290155 1330 raft_consensus.cc:2454] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b507c27af3ad4da499539f0621f698f5 in term 0.
I20260928 15:52:01.290151 1601 raft_consensus.cc:2454] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b507c27af3ad4da499539f0621f698f5 in term 0.
I20260928 15:52:01.290891 1398 leader_election.cc:302] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64fd823a7656431898d2f85305fa81cb, 9734394e456347eb957c4c742fb3646e, b507c27af3ad4da499539f0621f698f5; no voters:
I20260928 15:52:01.291380 1672 raft_consensus.cc:2781] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:52:01.291558 1672 raft_consensus.cc:482] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:52:01.291682 1672 raft_consensus.cc:3037] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:52:01.294183 1672 raft_consensus.cc:504] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } }
I20260928 15:52:01.294916 1672 leader_election.cc:288] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [CANDIDATE]: Term 1 election: Requested vote from peers 64fd823a7656431898d2f85305fa81cb (127.22.157.195:34583), 9734394e456347eb957c4c742fb3646e (127.22.157.193:44967)
I20260928 15:52:01.295367 1601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428eaa12c6c94787a2bf72414329e0ee" candidate_uuid: "b507c27af3ad4da499539f0621f698f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64fd823a7656431898d2f85305fa81cb"
I20260928 15:52:01.295383 1330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "428eaa12c6c94787a2bf72414329e0ee" candidate_uuid: "b507c27af3ad4da499539f0621f698f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9734394e456347eb957c4c742fb3646e"
I20260928 15:52:01.295589 1601 raft_consensus.cc:3037] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:52:01.295600 1330 raft_consensus.cc:3037] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:52:01.297996 1601 raft_consensus.cc:2454] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b507c27af3ad4da499539f0621f698f5 in term 1.
I20260928 15:52:01.298153 1330 raft_consensus.cc:2454] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b507c27af3ad4da499539f0621f698f5 in term 1.
I20260928 15:52:01.298511 1401 leader_election.cc:302] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64fd823a7656431898d2f85305fa81cb, b507c27af3ad4da499539f0621f698f5; no voters:
I20260928 15:52:01.298841 1672 raft_consensus.cc:2781] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:52:01.299311 1672 raft_consensus.cc:686] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 1 LEADER]: Becoming Leader. State: Replica: b507c27af3ad4da499539f0621f698f5, State: Running, Role: LEADER
I20260928 15:52:01.299803 1672 consensus_queue.cc:237] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } }
I20260928 15:52:01.304563 1184 catalog_manager.cc:5495] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b507c27af3ad4da499539f0621f698f5 (127.22.157.194). New cstate: current_term: 1 leader_uuid: "b507c27af3ad4da499539f0621f698f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b507c27af3ad4da499539f0621f698f5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35229 } health_report { overall_health: HEALTHY } } }
W20260928 15:52:01.311803 1648 tablet.cc:2369] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:52:01.342432 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.336660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.22.157.194:35229)
0928 15:52:01.336959 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.336963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.336985 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:52:01.337246 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.337250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.337331 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.337494 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:52:01.337928 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.338694 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.340108 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.340526 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.340628 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.341221 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:52:01.341762 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:52:01.341881 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.342028 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.342100 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 15:52:01.350834 1601 raft_consensus.cc:1261] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [term 1 FOLLOWER]: Refusing update from remote peer b507c27af3ad4da499539f0621f698f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:52:01.350834 1330 raft_consensus.cc:1261] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [term 1 FOLLOWER]: Refusing update from remote peer b507c27af3ad4da499539f0621f698f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:52:01.351866 1677 consensus_queue.cc:1035] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64fd823a7656431898d2f85305fa81cb" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 34583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:52:01.352298 1672 consensus_queue.cc:1035] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9734394e456347eb957c4c742fb3646e" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 44967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:52:01.366750 1681 mvcc.cc:205] Tried to move back new op lower bound from 7334341514642436096 to 7334341514448257024. Current Snapshot: MvccSnapshot[applied={T|T < 7334341514642436096}]
I20260928 15:52:01.379696 1184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:52:01.379994 1184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:52:01.383843 1184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97c8c68bc269468c914b40c9371d0fa0: Sending DeleteTablet for 3 replicas of tablet 428eaa12c6c94787a2bf72414329e0ee
I20260928 15:52:01.385298 1309 tablet_service.cc:1514] Processing DeleteTablet for tablet 428eaa12c6c94787a2bf72414329e0ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:52:01 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:33488
I20260928 15:52:01.385421 1446 tablet_service.cc:1514] Processing DeleteTablet for tablet 428eaa12c6c94787a2bf72414329e0ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:52:01 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:37870
I20260928 15:52:01.386219 1581 tablet_service.cc:1514] Processing DeleteTablet for tablet 428eaa12c6c94787a2bf72414329e0ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:52:01 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:46768
I20260928 15:52:01.387683 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 1246
I20260928 15:52:01.387717 1691 tablet_replica.cc:330] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: stopping tablet replica
I20260928 15:52:01.388239 1691 raft_consensus.cc:2229] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:52:01.389137 1691 raft_consensus.cc:2258] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:52:01.396080 1692 tablet_replica.cc:330] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: stopping tablet replica
I20260928 15:52:01.396711 1692 raft_consensus.cc:2229] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:52:01.397392 1692 raft_consensus.cc:2258] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:52:01.397708 1691 ts_tablet_manager.cc:1905] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:52:01.399950 1692 ts_tablet_manager.cc:1905] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:52:01.406306 1691 ts_tablet_manager.cc:1918] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:52:01.406579 1691 log.cc:1196] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/428eaa12c6c94787a2bf72414329e0ee
I20260928 15:52:01.407248 1691 ts_tablet_manager.cc:1939] T 428eaa12c6c94787a2bf72414329e0ee P b507c27af3ad4da499539f0621f698f5: Deleting consensus metadata
I20260928 15:52:01.408210 1692 ts_tablet_manager.cc:1918] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:52:01.408466 1692 log.cc:1196] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/428eaa12c6c94787a2bf72414329e0ee
I20260928 15:52:01.409132 1692 ts_tablet_manager.cc:1939] T 428eaa12c6c94787a2bf72414329e0ee P 64fd823a7656431898d2f85305fa81cb: Deleting consensus metadata
Sep 28 15:52:01 dist-test-slave-1vw0 krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:52:01.409827 1172 catalog_manager.cc:4841] TS b507c27af3ad4da499539f0621f698f5 (127.22.157.194:35229): tablet 428eaa12c6c94787a2bf72414329e0ee (table test-table [id=b90adf09597e49e89fecdd5d2390ffa3]) successfully deleted
I20260928 15:52:01.412019 1690 tablet_replica.cc:330] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e: stopping tablet replica
I20260928 15:52:01.412099 1172 catalog_manager.cc:4841] TS 64fd823a7656431898d2f85305fa81cb (127.22.157.195:34583): tablet 428eaa12c6c94787a2bf72414329e0ee (table test-table [id=b90adf09597e49e89fecdd5d2390ffa3]) successfully deleted
I20260928 15:52:01.412523 1690 raft_consensus.cc:2229] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:52:01.413136 1690 raft_consensus.cc:2258] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:52:01.415406 1690 ts_tablet_manager.cc:1905] T 428eaa12c6c94787a2bf72414329e0ee P 9734394e456347eb957c4c742fb3646e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:52:01.419109 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.389939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.22.157.193:44967)
0928 15:52:01.390138 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.390145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.394860 (+ 4715us) server_negotiation.cc:407] Connection header received
0928 15:52:01.395055 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.395065 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.395190 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.395359 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:01.398945 (+ 3586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.400137 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.404330 (+ 4193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.405050 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.405174 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.411024 (+ 5850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:01.411044 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:01.411063 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:01.411123 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:01.415423 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.416478 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.416488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.416492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.416691 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.417162 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.417173 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.417177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.417688 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:01.418067 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.418399 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.418665 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 15:52:01.419925 1371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:52:01.521512 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 1246
W20260928 15:52:01.540427 1169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.157.193:44967 (error 108)
I20260928 15:52:01.540707 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 1382
I20260928 15:52:01.542616 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.542201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.0.0.1:33516)
0928 15:52:01.542361 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.542472 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:52:01.542881 1169 catalog_manager.cc:4584] TS 9734394e456347eb957c4c742fb3646e (127.22.157.193:44967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 428eaa12c6c94787a2bf72414329e0ee: Network error: Client connection negotiation failed: client connection to 127.22.157.193:44967: connect: Connection refused (error 111)
Sep 28 15:52:01 dist-test-slave-1vw0 krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:52:01.559373 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.541585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.22.157.194:35229)
0928 15:52:01.541773 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.541778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.541938 (+ 160us) server_negotiation.cc:407] Connection header received
0928 15:52:01.542166 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.542172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.542296 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.542398 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:01.544217 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.545287 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.547171 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.547601 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.547694 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.553136 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:01.553156 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:01.553175 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:01.553217 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:01.556308 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.557130 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.557138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.557140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.557307 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.557694 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.557705 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.557713 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.558100 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:01.558451 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.558749 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.558999 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:52:01.560215 1507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:52:01.596563 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.596047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.0.0.1:33522)
0928 15:52:01.596277 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.596396 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 15:52:01.597733 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.597343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.0.0.1:33524)
0928 15:52:01.597498 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.597619 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 15:52:01.658524 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.657971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.0.0.1:33538)
0928 15:52:01.658213 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.658305 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 15:52:01.659713 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.659309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.0.0.1:33552)
0928 15:52:01.659486 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.659567 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 15:52:01.678923 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 1382
I20260928 15:52:01.698150 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 1518
Sep 28 15:52:01 dist-test-slave-1vw0 krb5kdc[1140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:52:01.714483 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.698946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.22.157.195:34583)
0928 15:52:01.699128 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:01.699136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:01.699195 (+ 59us) server_negotiation.cc:407] Connection header received
0928 15:52:01.699426 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:01.699431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:01.699496 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:01.699657 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:01.700981 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.701722 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.703046 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:01.703472 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:01.703620 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:01.708430 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:01.708447 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:01.708462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:01.708512 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:01.711279 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.712074 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.712083 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.712085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.712246 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:01.712669 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:01.712674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:01.712679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:01.713223 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:01.713536 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:01.713877 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 15:52:01.714111 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:52:01.715235 1642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:52:01.740451 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.739828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.0.0.1:33566)
0928 15:52:01.740102 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.740221 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 15:52:01.741688 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.741229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.0.0.1:33576)
0928 15:52:01.741448 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.741538 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 15:52:01.809685 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 1518
I20260928 15:52:01.826699 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 1153
I20260928 15:52:01.828027 1215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:52:01.939963 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:01.939229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:44967 (local address 127.0.0.1:33588)
0928 15:52:01.939500 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:01.939624 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:44967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 15:52:01.940099 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 1153
2026-09-28T15:52:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4532 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:52:02 dist-test-slave-1vw0 krb5kdc[1701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:52:02 dist-test-slave-1vw0 krb5kdc[1701](info): set up 2 sockets
Sep 28 15:52:02 dist-test-slave-1vw0 krb5kdc[1701](info): commencing operation
krb5kdc: starting...
W20260928 15:52:03.058151 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:52:03 dist-test-slave-1vw0 krb5kdc[1701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:52:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/rangeradmin_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/rangeradmin_127.22.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/rangerlookup_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/rangerlookup_127.22.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/HTTP_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/HTTP_127.22.157.212@KRBTEST.COM.keytab.
I20260928 15:52:03.435708 23159 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1s6B80/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/postgres -l logfile start
2026-09-28 15:52:06.115 UTC [1731] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 15:52:06.115 UTC [1731] LOG: listening on IPv4 address "127.22.157.212", port 40511
2026-09-28 15:52:06.119 UTC [1731] LOG: listening on Unix socket "/tmp/.s.PGSQL.40511"
2026-09-28 15:52:06.143 UTC [1734] LOG: database system was shut down at 2026-09-28 15:52:04 UTC
2026-09-28 15:52:06.147 UTC [1731] LOG: database system is ready to accept connections
I20260928 15:52:07.154078 23159 mini_postgres.cc:96] Postgres bound to 40511
2026-09-28 15:52:07.175 UTC [1742] FATAL: database "slave" does not exist
127.22.157.212:40511 - accepting connections
I20260928 15:52:07.175879 23159 mini_ranger.cc:161] Starting Ranger...
I20260928 15:52:07.206076 23159 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 15:52:07.728960 23159 mini_ranger.cc:87] Created ranger Postgres database
I20260928 15:52:07.729100 23159 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-admin
2026-09-28 15:52:08,084 [I] DB FLAVOR :POSTGRES
2026-09-28 15:52:08,085 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:52:08,085 [I] Checking connection..
2026-09-28 15:52:08,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 15:52:08,375 [I] Checking connection passed.
2026-09-28 15:52:08,376 [I] --------- Verifying version history table ---------
2026-09-28 15:52:08,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:52:08,630 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:52:08,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:52:08,882 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:52:08,883 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 15:52:08,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 15:52:09.117 UTC [1807] WARNING: there is no transaction in progress
2026-09-28 15:52:09,123 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 15:52:09,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:52:09,391 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 15:52:09,391 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:52:09,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 15:52:09,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 15:52:09,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1vw0.c.gcp-upstream.internal','N') ;"
2026-09-28 15:52:10,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:52:10,414 [I] Table x_portal_user does not exist in database ranger
2026-09-28 15:52:10,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:52:10,680 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 15:52:10,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:52:10,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:52:11,166 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 15:52:11,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 15:52:12.175 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.191 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.208 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.225 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.242 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.263 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.319 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.327 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.341 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.352 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.362 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.375 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.385 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.391 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.397 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:12.404 UTC [1917] WARNING: there is no transaction in progress
2026-09-28 15:52:13,088 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 15:52:13,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 15:52:13,327 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 15:52:13,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:52:13,599 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 15:52:13,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:52:13,839 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 15:52:13,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:52:14,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:52:14,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1vw0.c.gcp-upstream.internal';"
2026-09-28 15:52:14,589 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 15:52:14,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 15:52:14,857 [I] DB_PATCHES have already been applied
I20260928 15:52:14.863960 23159 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-admin:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1s6B80/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 15:52:14.864053 23159 mini_ranger.cc:193] Using host: 127.22.157.212
I20260928 15:52:14.882053 23159 mini_ranger.cc:227] Ranger admin URL: http://127.22.157.212:48635
Sep 28, 2026 3:52:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:52:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:52:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:52:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:52:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:52:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:52:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 15:52:16.032055 23159 mini_ranger.cc:160] Time spent starting Ranger: real 8.856s user 0.000s sys 0.050s
Sep 28, 2026 3:52:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:52:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 3:52:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:52:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:52:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 15:52:37.896100 23159 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/rangerkms_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/rangerkms_127.22.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.157.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.157.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.157.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/HTTP_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.157.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/HTTP_127.22.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 15:52:38 dist-test-slave-1vw0 krb5kdc[1701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610758, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 15:52:38.096196 23159 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 15:52:38.122416 23159 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 15:52:39.203989 23159 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 15:52:39.204164 23159 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-kms
I20260928 15:52:39.204311 23159 mini_ranger_kms.cc:227] Using postgres at 127.22.157.212:40511
2026-09-28 15:52:39,867 [I] DB FLAVOR :POSTGRES
2026-09-28 15:52:39,867 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:52:39,867 [I] Checking connection
2026-09-28 15:52:39,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 15:52:40,103 [I] connection success
2026-09-28 15:52:40,104 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 15:52:40,104 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 15:52:40,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 15:52:40,370 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 15:52:40,370 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:52:40,370 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 15:52:40,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:40511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 15:52:40,639 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 15:52:41.158324 23159 mini_ranger_kms.cc:310] Created kms service
I20260928 15:52:41.409304 23159 mini_ranger_kms.cc:326] Created kudu user
I20260928 15:52:41.462131 23159 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 15:52:41 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610758, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:52:41.928047 23159 mini_ranger_kms.cc:384] Added ranger policy
I20260928 15:52:41.928387 23159 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-kms:/tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 15:52:41.928452 23159 mini_ranger_kms.cc:240] Using host: 127.22.157.212
I20260928 15:52:41.957729 23159 mini_ranger_kms.cc:276] Ranger KMS PID: 2099
I20260928 15:52:41.957933 23159 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.157.212:47707
Sep 28, 2026 3:52:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:52:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:52:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:52:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:52:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:52:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 15:52:44 dist-test-slave-1vw0 krb5kdc[1701](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610764, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.157.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:52:44.291865 23159 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.196s user 0.143s sys 0.397s
I20260928 15:52:44.292135 23159 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 15:52:44.292353 23159 mini_ranger_kms.cc:401] 127.22.157.212:47707/kms/v1/keys
Sep 28 15:52:44 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610764, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.157.212@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
Sep 28 15:52:45 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610764, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.157.212@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
Sep 28 15:52:47 dist-test-slave-1vw0 krb5kdc[1701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:52:47.569545 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:46803
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:46803
--ranger_config_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:47707/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:52:47.955452 2150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:52:47.955780 2150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:52:47.955873 2150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:52:47.963557 2150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:52:47.963696 2150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:52:47.963743 2150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:52:47.963793 2150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:52:47.963843 2150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:52:47.974421 2150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.157.212:47707/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:46803
--ranger_config_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:46803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:52:47.975772 2150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:52:47.977177 2150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:52:47.990674 2158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:47.990734 2155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:47.990743 2156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:52:47.991958 2150 server_base.cc:1034] running on GCE node
I20260928 15:52:47.992861 2150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:52:47.994433 2150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:52:47.995649 2150 hybrid_clock.cc:648] HybridClock initialized: now 1790610767995578 us; error 58 us; skew 500 ppm
I20260928 15:52:47.995997 2150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:52:47 dist-test-slave-1vw0 krb5kdc[1701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610767, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:52:47.999359 2150 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:52:48.001003 2150 webserver.cc:460] Webserver started at http://127.22.157.254:44899/ using document root <none> and password file <none>
I20260928 15:52:48.001694 2150 fs_manager.cc:359] Metadata directory not provided
I20260928 15:52:48.001825 2150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:52:48.002162 2150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:52:48 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610767, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:52:48.058050 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "e857370b1f1d42ba9d1ec6ec939746b6"
format_stamp: "Formatted at 2026-09-28 15:52:48 on dist-test-slave-1vw0"
server_key: "6d48f85591d78f9517c7422270baf0e2"
server_key_iv: "43e6ddb1ca82b633caaf96248563cbfc"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:48.059003 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "e857370b1f1d42ba9d1ec6ec939746b6"
format_stamp: "Formatted at 2026-09-28 15:52:48 on dist-test-slave-1vw0"
server_key: "6d48f85591d78f9517c7422270baf0e2"
server_key_iv: "43e6ddb1ca82b633caaf96248563cbfc"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:48.065353 2150 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260928 15:52:48.087088 2165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:48.088987 2150 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:52:48.089205 2150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "e857370b1f1d42ba9d1ec6ec939746b6"
format_stamp: "Formatted at 2026-09-28 15:52:48 on dist-test-slave-1vw0"
server_key: "6d48f85591d78f9517c7422270baf0e2"
server_key_iv: "43e6ddb1ca82b633caaf96248563cbfc"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:48.089453 2150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:52:48.117064 2150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:52:48.121726 2150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:52:48.122102 2150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:52:48.142381 2150 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:46803
I20260928 15:52:48.142401 2217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:46803 every 8 connection(s)
I20260928 15:52:48.143882 2150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:52:48.148152 2218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:52:48.152649 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 2150
I20260928 15:52:48.152925 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:52:48.153844 23159 external_mini_cluster.cc:1428] Setting key 4762d27fbbfda5bf3ded68085a90dac8
I20260928 15:52:48.164996 2218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6: Bootstrap starting.
Sep 28 15:52:48 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:52:48.168907 2218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6: Neither blocks nor log segments found. Creating new log.
I20260928 15:52:48.170300 2218 log.cc:824] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6: Log is configured to *not* fsync() on all Append() calls
I20260928 15:52:48.173905 2218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6: No bootstrap required, opened a new log
I20260928 15:52:48.178503 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:48.156326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.22.157.254:46803)
0928 15:52:48.157242 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:48.157255 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:48.157299 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:52:48.158950 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:48.158977 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:48.159416 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:48.159869 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:48.161305 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:48.162730 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:48.164098 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:48.165037 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:48.165232 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:48.169916 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:48.169925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:48.169942 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:48.169989 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:48.173653 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:48.174653 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:48.174662 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:48.174669 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:48.174850 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:48.175309 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:48.175312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:48.175314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:48.175920 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:48.176167 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:48.176854 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 15:52:48.177345 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":482,"threads_started":1}
I20260928 15:52:48.182801 2218 raft_consensus.cc:348] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e857370b1f1d42ba9d1ec6ec939746b6" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46803 } }
I20260928 15:52:48.183228 2218 raft_consensus.cc:374] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:52:48.183360 2218 raft_consensus.cc:729] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e857370b1f1d42ba9d1ec6ec939746b6, State: Initialized, Role: FOLLOWER
I20260928 15:52:48.183979 2218 consensus_queue.cc:260] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e857370b1f1d42ba9d1ec6ec939746b6" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46803 } }
I20260928 15:52:48.184231 2218 raft_consensus.cc:388] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:52:48.184377 2218 raft_consensus.cc:482] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:52:48.184553 2218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:52:48.186880 2218 raft_consensus.cc:504] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e857370b1f1d42ba9d1ec6ec939746b6" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46803 } }
I20260928 15:52:48.187403 2218 leader_election.cc:302] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e857370b1f1d42ba9d1ec6ec939746b6; no voters:
I20260928 15:52:48.188011 2218 leader_election.cc:288] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:52:48.188160 2223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:52:48.188956 2223 raft_consensus.cc:686] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [term 1 LEADER]: Becoming Leader. State: Replica: e857370b1f1d42ba9d1ec6ec939746b6, State: Running, Role: LEADER
I20260928 15:52:48.189565 2223 consensus_queue.cc:237] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e857370b1f1d42ba9d1ec6ec939746b6" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46803 } }
I20260928 15:52:48.190228 2218 sys_catalog.cc:564] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:52:48.197826 2224 sys_catalog.cc:455] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e857370b1f1d42ba9d1ec6ec939746b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e857370b1f1d42ba9d1ec6ec939746b6" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46803 } } }
I20260928 15:52:48.197810 2225 sys_catalog.cc:455] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e857370b1f1d42ba9d1ec6ec939746b6. Latest consensus state: current_term: 1 leader_uuid: "e857370b1f1d42ba9d1ec6ec939746b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e857370b1f1d42ba9d1ec6ec939746b6" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46803 } } }
I20260928 15:52:48.198525 2224 sys_catalog.cc:458] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:52:48.198570 2225 sys_catalog.cc:458] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6 [sys.catalog]: This master's current role is: LEADER
I20260928 15:52:48.199558 2230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:52:48.206213 2230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:52:48.206938 2218 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:52:48.226356 2230 catalog_manager.cc:1347] Generated new cluster ID: 161e46f7d52747ec857657edebc04b30
I20260928 15:52:48.226526 2230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:52:48.241420 2230 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:52:48.243569 2230 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:52:48.264446 2230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6: Generated new TSK 0
I20260928 15:52:48.265719 2230 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:52:48.973695 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:47707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46803
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:52:49 dist-test-slave-1vw0 krb5kdc[1701](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610769, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 15:52:49.411713 2263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:52:49.412144 2263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:52:49.412359 2263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:52:49.423854 2263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:52:49.423978 2263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:52:49.424171 2263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:52:49.440114 2263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.157.212:47707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:52:49.442011 2263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:52:49.443892 2263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:52:49.464258 2274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:49.465205 2275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:49.471432 2277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:52:49.471777 2263 server_base.cc:1034] running on GCE node
I20260928 15:52:49.472858 2263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:52:49.474181 2263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:52:49.475517 2263 hybrid_clock.cc:648] HybridClock initialized: now 1790610769475432 us; error 68 us; skew 500 ppm
I20260928 15:52:49.475907 2263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:52:49 dist-test-slave-1vw0 krb5kdc[1701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610769, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:52:49.480284 2263 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:52:49.484591 2263 webserver.cc:460] Webserver started at http://127.22.157.193:35661/ using document root <none> and password file <none>
I20260928 15:52:49.485623 2263 fs_manager.cc:359] Metadata directory not provided
I20260928 15:52:49.485786 2263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:52:49.486189 2263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:52:49 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610769, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:52:49.509742 2263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "c15b180b33d141a8a01b59d91b06f4db"
format_stamp: "Formatted at 2026-09-28 15:52:49 on dist-test-slave-1vw0"
server_key: "38554f304f33f984ef9a0e979b23dedb"
server_key_iv: "bb521fd1e1f2220929ffcb361097a641"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:49.510983 2263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "c15b180b33d141a8a01b59d91b06f4db"
format_stamp: "Formatted at 2026-09-28 15:52:49 on dist-test-slave-1vw0"
server_key: "38554f304f33f984ef9a0e979b23dedb"
server_key_iv: "bb521fd1e1f2220929ffcb361097a641"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:49.531266 2263 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.006s sys 0.011s
I20260928 15:52:49.566067 2284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:49.567862 2263 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.007s
I20260928 15:52:49.568080 2263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "c15b180b33d141a8a01b59d91b06f4db"
format_stamp: "Formatted at 2026-09-28 15:52:49 on dist-test-slave-1vw0"
server_key: "38554f304f33f984ef9a0e979b23dedb"
server_key_iv: "bb521fd1e1f2220929ffcb361097a641"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:49.568334 2263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:52:49.605420 2263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:52:49.610450 2263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:52:49.610803 2263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:52:49.611910 2263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:52:49.613590 2263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:52:49.613703 2263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:49.613864 2263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:52:49.613947 2263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:49.678558 2263 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:42319
I20260928 15:52:49.678586 2396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:42319 every 8 connection(s)
I20260928 15:52:49.680150 2263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:52:49.685526 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 2263
I20260928 15:52:49.685791 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:52:49.686815 23159 external_mini_cluster.cc:1428] Setting key 127f651a6519d3aec5b024bdb109f4f1
Sep 28 15:52:49 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610769, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
I20260928 15:52:49.737361 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:49.687735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:41347 (local address 127.22.157.254:46803)
0928 15:52:49.708512 (+ 20777us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:49.708520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:49.709787 (+ 1267us) server_negotiation.cc:407] Connection header received
0928 15:52:49.712815 (+ 3028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:49.712825 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:49.712957 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:49.713054 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:49.717705 (+ 4651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:49.718942 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:49.723007 (+ 4065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:49.723704 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:49.723817 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:49.729093 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:49.729102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:49.729107 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:49.729165 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:49.732940 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:49.734022 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:49.734028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:49.734032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:49.734221 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:49.735061 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:49.735068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:49.735072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:49.735464 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:49.735618 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:49.736682 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0928 15:52:49.736963 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20606,"thread_start_us":4358,"threads_started":1}
I20260928 15:52:49.740206 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:49.684607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46803 (local address 127.22.157.193:41347)
0928 15:52:49.709568 (+ 24961us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:49.709638 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:49.712269 (+ 2631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:49.713758 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:49.713772 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:49.714474 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:52:49.715370 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:49.715395 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:49.720132 (+ 4737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:49.720139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:49.722607 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:49.722639 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:49.724426 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:49.724435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:49.724833 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:49.725747 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:52:49.725781 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:52:49.728787 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:52:49.733146 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:49.733156 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:49.733177 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:49.733847 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:49.734608 (+ 761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:49.734614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:49.734618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:49.734910 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:49.735718 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:52:49.735726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:52:49.736220 (+ 494us) client_negotiation.cc:769] Sending connection context
0928 15:52:49.737947 (+ 1727us) client_negotiation.cc:240] Negotiation successful
0928 15:52:49.738588 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24550,"thread_start_us":19695,"threads_started":1}
I20260928 15:52:49.742825 2397 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46803
I20260928 15:52:49.743211 2397 heartbeater.cc:464] Registering TS with master...
I20260928 15:52:49.744146 2397 heartbeater.cc:511] Master 127.22.157.254:46803 requested a full tablet report, sending...
I20260928 15:52:49.746707 2182 ts_manager.cc:194] Registered new tserver with Master: c15b180b33d141a8a01b59d91b06f4db (127.22.157.193:42319)
I20260928 15:52:49.748641 2182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:41347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:52:49.772943 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:47707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46803
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:52:49 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610769, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
Sep 28 15:52:50 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610769, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
W20260928 15:52:50.347734 2404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:52:50.348289 2404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:52:50.348522 2404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:52:50.362080 2404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:52:50.362322 2404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:52:50.362588 2404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:52:50.379865 2404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.157.212:47707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:52:50.382208 2404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:52:50.384405 2404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:52:50.423069 2412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:50.426795 2411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:52:50.435880 2404 server_base.cc:1034] running on GCE node
W20260928 15:52:50.437072 2414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:52:50.438182 2404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:52:50.444846 2404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:52:50.448375 2404 hybrid_clock.cc:648] HybridClock initialized: now 1790610770448229 us; error 131 us; skew 500 ppm
I20260928 15:52:50.448962 2404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:52:50 dist-test-slave-1vw0 krb5kdc[1701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610770, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:52:50.455371 2404 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:52:50.470216 2404 webserver.cc:460] Webserver started at http://127.22.157.194:36329/ using document root <none> and password file <none>
I20260928 15:52:50.471180 2404 fs_manager.cc:359] Metadata directory not provided
I20260928 15:52:50.471338 2404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:52:50.471715 2404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:52:50 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610770, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:52:50.502113 2404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "24c97562a33544a29a248018b453b8e5"
format_stamp: "Formatted at 2026-09-28 15:52:50 on dist-test-slave-1vw0"
server_key: "7d0e50869f752530a92b51eff557b6cc"
server_key_iv: "381243a3bb68dc8b03c3a5a0cb481d69"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:50.504112 2404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "24c97562a33544a29a248018b453b8e5"
format_stamp: "Formatted at 2026-09-28 15:52:50 on dist-test-slave-1vw0"
server_key: "7d0e50869f752530a92b51eff557b6cc"
server_key_iv: "381243a3bb68dc8b03c3a5a0cb481d69"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:50.528450 2404 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.010s sys 0.011s
I20260928 15:52:50.573966 2428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:50.576526 2404 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.011s sys 0.007s
I20260928 15:52:50.576926 2404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "24c97562a33544a29a248018b453b8e5"
format_stamp: "Formatted at 2026-09-28 15:52:50 on dist-test-slave-1vw0"
server_key: "7d0e50869f752530a92b51eff557b6cc"
server_key_iv: "381243a3bb68dc8b03c3a5a0cb481d69"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:50.577425 2404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:52:50.641742 2404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:52:50.647998 2404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:52:50.648473 2404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:52:50.656276 2404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:52:50.665964 2404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:52:50.666267 2404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:52:50.666566 2404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:52:50.666735 2404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:50.752832 2397 heartbeater.cc:503] Master 127.22.157.254:46803 was elected leader, sending a full tablet report...
I20260928 15:52:50.812964 2404 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:37329
I20260928 15:52:50.812983 2540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:37329 every 8 connection(s)
I20260928 15:52:50.814859 2404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:52:50.816772 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 2404
I20260928 15:52:50.817010 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:52:50.817857 23159 external_mini_cluster.cc:1428] Setting key 57247aacb55f0f1a83017bc5df7d9ce6
Sep 28 15:52:50 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610770, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
I20260928 15:52:50.869964 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:50.833862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:55965 (local address 127.22.157.254:46803)
0928 15:52:50.842161 (+ 8299us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:50.842172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:50.846399 (+ 4227us) server_negotiation.cc:407] Connection header received
0928 15:52:50.846481 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:50.846490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:50.846599 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:50.846711 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:50.849488 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:50.850809 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:50.853839 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:50.854631 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:50.854762 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:50.860140 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:50.860150 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:50.860159 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:50.860216 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:50.864649 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:50.866403 (+ 1754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:50.866408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:50.866410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:50.866730 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:50.867487 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:50.867492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:50.867495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:50.867883 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:50.868220 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:50.869200 (+ 980us) server_negotiation.cc:299] Negotiation successful
0928 15:52:50.869510 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8059,"thread_start_us":7809,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
I20260928 15:52:50.871516 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:50.832774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46803 (local address 127.22.157.194:55965)
0928 15:52:50.843542 (+ 10768us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:50.843603 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:50.845971 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:50.847280 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:50.847298 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:50.848197 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:52:50.849222 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:50.849259 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:50.851047 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:50.851055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:50.853629 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:50.853651 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:50.855161 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:50.855170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:50.855524 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:50.856592 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:52:50.856630 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:52:50.859770 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:52:50.865003 (+ 5233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:50.865018 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:50.865043 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:50.866210 (+ 1167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:50.866908 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:50.866915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:50.866919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:50.867282 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:50.868166 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:52:50.868176 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:52:50.868621 (+ 445us) client_negotiation.cc:769] Sending connection context
0928 15:52:50.869023 (+ 402us) client_negotiation.cc:240] Negotiation successful
0928 15:52:50.869693 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10390,"thread_start_us":9882,"threads_started":1}
I20260928 15:52:50.874329 2541 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46803
I20260928 15:52:50.874816 2541 heartbeater.cc:464] Registering TS with master...
I20260928 15:52:50.875844 2541 heartbeater.cc:511] Master 127.22.157.254:46803 requested a full tablet report, sending...
I20260928 15:52:50.877938 2182 ts_manager.cc:194] Registered new tserver with Master: 24c97562a33544a29a248018b453b8e5 (127.22.157.194:37329)
I20260928 15:52:50.879352 2182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:55965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:52:50.908249 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:47707/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46803
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:52:51.332778 2548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:52:51.333106 2548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:52:51.333298 2548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:52:51.341928 2548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:52:51.342052 2548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:52:51.342221 2548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:52:51.354103 2548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.157.212:47707/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:52:51.356040 2548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:52:51.357748 2548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:52:51.376284 2553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:51.376142 2556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:52:51.376412 2554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:52:51.377204 2548 server_base.cc:1034] running on GCE node
I20260928 15:52:51.377909 2548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:52:51.378931 2548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:52:51.380179 2548 hybrid_clock.cc:648] HybridClock initialized: now 1790610771380106 us; error 61 us; skew 500 ppm
I20260928 15:52:51.380532 2548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:52:51 dist-test-slave-1vw0 krb5kdc[1701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610771, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:52:51.383788 2548 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:52:51.385489 2548 webserver.cc:460] Webserver started at http://127.22.157.195:37123/ using document root <none> and password file <none>
I20260928 15:52:51.386257 2548 fs_manager.cc:359] Metadata directory not provided
I20260928 15:52:51.386404 2548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:52:51.386749 2548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:52:51 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610771, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:52:51.405360 2548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "aa8591298b20482ca83420aab2e8c25b"
format_stamp: "Formatted at 2026-09-28 15:52:51 on dist-test-slave-1vw0"
server_key: "24009bf5022dff60c08fd016526636a0"
server_key_iv: "8c617be1584a14ab58151f479d2e5ba9"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:51.406286 2548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "aa8591298b20482ca83420aab2e8c25b"
format_stamp: "Formatted at 2026-09-28 15:52:51 on dist-test-slave-1vw0"
server_key: "24009bf5022dff60c08fd016526636a0"
server_key_iv: "8c617be1584a14ab58151f479d2e5ba9"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:51.412514 2548 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 15:52:51.428246 2563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:51.429991 2548 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:52:51.430171 2548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "aa8591298b20482ca83420aab2e8c25b"
format_stamp: "Formatted at 2026-09-28 15:52:51 on dist-test-slave-1vw0"
server_key: "24009bf5022dff60c08fd016526636a0"
server_key_iv: "8c617be1584a14ab58151f479d2e5ba9"
server_key_version: "kuduclusterkey@0"
I20260928 15:52:51.430408 2548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:52:51.467196 2548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:52:51.471688 2548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:52:51.472092 2548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:52:51.473397 2548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:52:51.475418 2548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:52:51.475524 2548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:51.475678 2548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:52:51.475749 2548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:52:51.528286 2548 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:43437
I20260928 15:52:51.528321 2675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:43437 every 8 connection(s)
I20260928 15:52:51.529731 2548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:52:51.530557 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 2548
I20260928 15:52:51.530795 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:52:51.531412 23159 external_mini_cluster.cc:1428] Setting key 0e2ab1df2807d54aeaa5fa3c784c1c8a
Sep 28 15:52:51 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610771, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:52:51.555233 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.533013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:38521 (local address 127.22.157.254:46803)
0928 15:52:51.533639 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:51.533663 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:51.536123 (+ 2460us) server_negotiation.cc:407] Connection header received
0928 15:52:51.538715 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:51.538720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:51.538841 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:51.538948 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:51.541180 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.541909 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.543658 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.544109 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.544214 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.548926 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:51.548932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:51.548936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:51.548989 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:51.551848 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:51.552702 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:51.552707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:51.552710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:51.552896 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:51.553385 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:51.553390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:51.553393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:51.553704 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:51.553875 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:51.554611 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 15:52:51.554851 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":341,"threads_started":1}
I20260928 15:52:51.556315 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.534119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46803 (local address 127.22.157.195:38521)
0928 15:52:51.535884 (+ 1765us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:51.535989 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:51.538281 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:51.539313 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:51.539330 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:51.540092 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:52:51.540975 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.540997 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.542101 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.542106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.543495 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.543509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.544374 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.544419 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.544691 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.545783 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:52:51.545848 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:52:51.548621 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:52:51.552036 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:51.552047 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:51.552068 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:51.552559 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:51.553014 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:52:51.553019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:52:51.553023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:52:51.553258 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:52:51.553837 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:52:51.553845 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:52:51.554242 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 15:52:51.554518 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 15:52:51.555117 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":541,"threads_started":1}
I20260928 15:52:51.558104 2676 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46803
I20260928 15:52:51.558532 2676 heartbeater.cc:464] Registering TS with master...
I20260928 15:52:51.559967 2676 heartbeater.cc:511] Master 127.22.157.254:46803 requested a full tablet report, sending...
I20260928 15:52:51.561712 2182 ts_manager.cc:194] Registered new tserver with Master: aa8591298b20482ca83420aab2e8c25b (127.22.157.195:43437)
I20260928 15:52:51.562578 2182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:38521
I20260928 15:52:51.574148 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:52:51.607677 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.579020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57246 (local address 127.22.157.254:46803)
0928 15:52:51.579195 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:51.579200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:51.579760 (+ 560us) server_negotiation.cc:407] Connection header received
0928 15:52:51.579970 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:51.579976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:51.580069 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:51.580181 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:51.581709 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.582553 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.583970 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.584545 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.584700 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.598728 (+ 14028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:51.598734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:51.598738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:51.598784 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:51.602846 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:51.603836 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:51.603842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:51.603844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:51.604103 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:51.604717 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:51.604782 (+ 65us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:51.604789 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:51.605373 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:51.605819 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:51.606229 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 15:52:51.606823 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:52:51.612545 2182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:52:51.620770 2182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:52:51.665472 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.636989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:37329 (local address 127.0.0.1:34606)
0928 15:52:51.649686 (+ 12697us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:51.649739 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:51.649931 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:51.657147 (+ 7216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:51.657152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:51.657179 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:51.657687 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.657705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.659605 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.659611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.662354 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.662375 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.664612 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.664618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.664822 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.664851 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:52:51.664928 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 15:52:51.665050 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12368,"mutex_wait_us":22,"thread_start_us":17457,"threads_started":1}
I20260928 15:52:51.665496 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.637148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:43437 (local address 127.0.0.1:49862)
0928 15:52:51.654764 (+ 17616us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:51.654785 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:51.654926 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:51.656139 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:51.656147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:51.656187 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:51.656669 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.656682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.658707 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.658716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.662436 (+ 3720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.662454 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.664274 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.664280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.664474 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.664550 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 15:52:51.664691 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 15:52:51.664956 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17462,"mutex_wait_us":43,"thread_start_us":13027,"threads_started":1}
I20260928 15:52:51.665612 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.637571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.22.157.195:43437)
0928 15:52:51.652052 (+ 14481us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:51.652062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:51.654928 (+ 2866us) server_negotiation.cc:407] Connection header received
0928 15:52:51.655112 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:51.655125 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:51.655507 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:51.655805 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:51.657131 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.658557 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.662674 (+ 4117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.664052 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.664411 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.664590 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:51.664805 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 15:52:51.665011 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14217,"thread_start_us":19150,"threads_started":1}
I20260928 15:52:51.666222 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.637033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.22.157.194:37329)
0928 15:52:51.656391 (+ 19358us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:51.656401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:51.656431 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:52:51.656594 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:51.656605 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:51.656911 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:51.657156 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:51.657907 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.659441 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.662567 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.664105 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.664266 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.664455 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:51.665771 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0928 15:52:51.665962 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19165,"thread_start_us":19829,"threads_started":1}
I20260928 15:52:51.667610 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.651303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42319 (local address 127.0.0.1:42912)
0928 15:52:51.655645 (+ 4342us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:51.655663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:51.655778 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:51.658970 (+ 3192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:51.658976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:51.659001 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:51.659516 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.659542 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.661864 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.661869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.663985 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.664001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.666958 (+ 2957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.666964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.667140 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.667179 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:52:51.667263 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 15:52:51.667372 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4224,"thread_start_us":4536,"threads_started":1}
I20260928 15:52:51.669480 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.652053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.22.157.193:42319)
0928 15:52:51.658125 (+ 6072us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:51.658135 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:51.658163 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:52:51.658262 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:51.658270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:51.658534 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:51.658728 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:51.659754 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.661695 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.664170 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.666725 (+ 2555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.666941 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.668863 (+ 1922us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:51.668938 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 15:52:51.669149 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5876,"thread_start_us":5636,"threads_started":1}
I20260928 15:52:51.670320 2611 tablet_service.cc:1467] Processing CreateTablet for tablet aa82b6cdb00945e2af1b4c3152cd92af (DEFAULT_TABLE table=test-table [id=e9a273e26f7f4ba5b221f38ff66a5759]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:52:51.670320 2476 tablet_service.cc:1467] Processing CreateTablet for tablet aa82b6cdb00945e2af1b4c3152cd92af (DEFAULT_TABLE table=test-table [id=e9a273e26f7f4ba5b221f38ff66a5759]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:52:51.672087 2611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa82b6cdb00945e2af1b4c3152cd92af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:52:51.672181 2476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa82b6cdb00945e2af1b4c3152cd92af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:52:51.674749 2332 tablet_service.cc:1467] Processing CreateTablet for tablet aa82b6cdb00945e2af1b4c3152cd92af (DEFAULT_TABLE table=test-table [id=e9a273e26f7f4ba5b221f38ff66a5759]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:52:51.676179 2332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa82b6cdb00945e2af1b4c3152cd92af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:52:51.688382 2696 tablet_bootstrap.cc:492] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: Bootstrap starting.
I20260928 15:52:51.689759 2697 tablet_bootstrap.cc:492] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: Bootstrap starting.
I20260928 15:52:51.692656 2696 tablet_bootstrap.cc:654] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: Neither blocks nor log segments found. Creating new log.
I20260928 15:52:51.693894 2697 tablet_bootstrap.cc:654] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: Neither blocks nor log segments found. Creating new log.
I20260928 15:52:51.694154 2698 tablet_bootstrap.cc:492] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: Bootstrap starting.
I20260928 15:52:51.694821 2696 log.cc:824] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:52:51.695354 2697 log.cc:824] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:52:51.698201 2697 tablet_bootstrap.cc:492] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: No bootstrap required, opened a new log
I20260928 15:52:51.698473 2696 tablet_bootstrap.cc:492] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: No bootstrap required, opened a new log
I20260928 15:52:51.698520 2698 tablet_bootstrap.cc:654] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: Neither blocks nor log segments found. Creating new log.
I20260928 15:52:51.698511 2697 ts_tablet_manager.cc:1397] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:52:51.698786 2696 ts_tablet_manager.cc:1397] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 15:52:51.700008 2698 log.cc:824] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: Log is configured to *not* fsync() on all Append() calls
I20260928 15:52:51.703590 2698 tablet_bootstrap.cc:492] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: No bootstrap required, opened a new log
I20260928 15:52:51.703896 2698 ts_tablet_manager.cc:1397] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 15:52:51.707043 2697 raft_consensus.cc:348] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } }
I20260928 15:52:51.707461 2697 raft_consensus.cc:374] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:52:51.707559 2697 raft_consensus.cc:729] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c97562a33544a29a248018b453b8e5, State: Initialized, Role: FOLLOWER
I20260928 15:52:51.708218 2697 consensus_queue.cc:260] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } }
I20260928 15:52:51.709427 2541 heartbeater.cc:503] Master 127.22.157.254:46803 was elected leader, sending a full tablet report...
I20260928 15:52:51.710310 2697 ts_tablet_manager.cc:1428] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 15:52:51.710770 2696 raft_consensus.cc:348] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } }
I20260928 15:52:51.711464 2696 raft_consensus.cc:374] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:52:51.711649 2696 raft_consensus.cc:729] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8591298b20482ca83420aab2e8c25b, State: Initialized, Role: FOLLOWER
I20260928 15:52:51.712529 2696 consensus_queue.cc:260] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } }
I20260928 15:52:51.718120 2698 raft_consensus.cc:348] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } }
I20260928 15:52:51.718871 2698 raft_consensus.cc:374] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:52:51.719030 2698 raft_consensus.cc:729] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15b180b33d141a8a01b59d91b06f4db, State: Initialized, Role: FOLLOWER
I20260928 15:52:51.719844 2676 heartbeater.cc:503] Master 127.22.157.254:46803 was elected leader, sending a full tablet report...
I20260928 15:52:51.719828 2698 consensus_queue.cc:260] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } }
I20260928 15:52:51.721204 2696 ts_tablet_manager.cc:1428] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: Time spent starting tablet: real 0.022s user 0.023s sys 0.000s
I20260928 15:52:51.721815 2698 ts_tablet_manager.cc:1428] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
W20260928 15:52:51.781680 2677 tablet.cc:2369] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:52:51.828121 2704 raft_consensus.cc:482] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:52:51.828438 2704 raft_consensus.cc:504] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } }
I20260928 15:52:51.830281 2704 leader_election.cc:288] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24c97562a33544a29a248018b453b8e5 (127.22.157.194:37329), aa8591298b20482ca83420aab2e8c25b (127.22.157.195:43437)
I20260928 15:52:51.838644 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.830722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:44789 (local address 127.22.157.195:43437)
0928 15:52:51.830973 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:51.830982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:51.832180 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 15:52:51.832384 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:51.832389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:51.832481 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:51.832569 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:51.833628 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.834661 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.836817 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.837622 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.837802 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.837893 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:51.838223 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 15:52:51.838361 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:52:51.838940 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.830724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:37329 (local address 127.22.157.193:46931)
0928 15:52:51.831900 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:51.831923 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:51.832083 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:51.832504 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:51.832512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:51.832553 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:51.833148 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.833160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.834574 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.834582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.836575 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.836588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.837846 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.837853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.838015 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.838057 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:52:51.838163 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:52:51.838338 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":54,"thread_start_us":605,"threads_started":1}
I20260928 15:52:51.838949 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.831095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:46931 (local address 127.22.157.194:37329)
0928 15:52:51.831392 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:51.831398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:51.832039 (+ 641us) server_negotiation.cc:407] Connection header received
0928 15:52:51.832290 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:51.832297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:51.832388 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:51.832460 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:52:51.833302 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.834449 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.836788 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.837602 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.837799 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.837886 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:51.838193 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 15:52:51.838339 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 15:52:51.838980 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.830828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:43437 (local address 127.22.157.193:44789)
0928 15:52:51.832056 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0928 15:52:51.832075 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:52:51.832235 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:52:51.832622 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:52:51.832627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:52:51.832666 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:52:51.833355 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.833371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.834787 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.834797 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.836642 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:52:51.836655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.837898 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.837905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:52:51.838075 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.838098 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:52:51.838196 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:52:51.838351 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":58,"thread_start_us":883,"threads_started":1}
I20260928 15:52:51.840107 2631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa82b6cdb00945e2af1b4c3152cd92af" candidate_uuid: "c15b180b33d141a8a01b59d91b06f4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa8591298b20482ca83420aab2e8c25b" is_pre_election: true
I20260928 15:52:51.840086 2496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa82b6cdb00945e2af1b4c3152cd92af" candidate_uuid: "c15b180b33d141a8a01b59d91b06f4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c97562a33544a29a248018b453b8e5" is_pre_election: true
I20260928 15:52:51.840636 2631 raft_consensus.cc:2454] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c15b180b33d141a8a01b59d91b06f4db in term 0.
I20260928 15:52:51.840822 2496 raft_consensus.cc:2454] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c15b180b33d141a8a01b59d91b06f4db in term 0.
I20260928 15:52:51.841440 2286 leader_election.cc:302] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa8591298b20482ca83420aab2e8c25b, c15b180b33d141a8a01b59d91b06f4db; no voters:
I20260928 15:52:51.841918 2704 raft_consensus.cc:2781] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:52:51.842069 2704 raft_consensus.cc:482] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:52:51.842190 2704 raft_consensus.cc:3037] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 0 FOLLOWER]: Advancing to term 1
W20260928 15:52:51.842384 2542 tablet.cc:2369] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:52:51.844551 2704 raft_consensus.cc:504] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } }
I20260928 15:52:51.845500 2704 leader_election.cc:288] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [CANDIDATE]: Term 1 election: Requested vote from peers 24c97562a33544a29a248018b453b8e5 (127.22.157.194:37329), aa8591298b20482ca83420aab2e8c25b (127.22.157.195:43437)
I20260928 15:52:51.846014 2496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa82b6cdb00945e2af1b4c3152cd92af" candidate_uuid: "c15b180b33d141a8a01b59d91b06f4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24c97562a33544a29a248018b453b8e5"
I20260928 15:52:51.846119 2631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa82b6cdb00945e2af1b4c3152cd92af" candidate_uuid: "c15b180b33d141a8a01b59d91b06f4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa8591298b20482ca83420aab2e8c25b"
I20260928 15:52:51.846295 2496 raft_consensus.cc:3037] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:52:51.846459 2631 raft_consensus.cc:3037] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:52:51.850338 2496 raft_consensus.cc:2454] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c15b180b33d141a8a01b59d91b06f4db in term 1.
I20260928 15:52:51.850510 2631 raft_consensus.cc:2454] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c15b180b33d141a8a01b59d91b06f4db in term 1.
I20260928 15:52:51.850929 2287 leader_election.cc:302] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24c97562a33544a29a248018b453b8e5, c15b180b33d141a8a01b59d91b06f4db; no voters:
I20260928 15:52:51.851300 2704 raft_consensus.cc:2781] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:52:51.851761 2704 raft_consensus.cc:686] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 1 LEADER]: Becoming Leader. State: Replica: c15b180b33d141a8a01b59d91b06f4db, State: Running, Role: LEADER
I20260928 15:52:51.852273 2704 consensus_queue.cc:237] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } }
I20260928 15:52:51.857712 2182 catalog_manager.cc:5495] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db reported cstate change: term changed from 0 to 1, leader changed from <none> to c15b180b33d141a8a01b59d91b06f4db (127.22.157.193). New cstate: current_term: 1 leader_uuid: "c15b180b33d141a8a01b59d91b06f4db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c15b180b33d141a8a01b59d91b06f4db" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42319 } health_report { overall_health: HEALTHY } } }
I20260928 15:52:51.894763 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.888634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.22.157.193:42319)
0928 15:52:51.888859 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:51.888864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:51.888882 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:52:51.888998 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:51.889002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:51.889090 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:51.889225 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:52:51.889588 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.890563 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.892225 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.892774 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.892898 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.893457 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:52:51.894122 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:52:51.894294 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:51.894352 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 15:52:51.894442 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:52:51.904955 2631 raft_consensus.cc:1261] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [term 1 FOLLOWER]: Refusing update from remote peer c15b180b33d141a8a01b59d91b06f4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:52:51.905004 2496 raft_consensus.cc:1261] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [term 1 FOLLOWER]: Refusing update from remote peer c15b180b33d141a8a01b59d91b06f4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:52:51.906206 2707 consensus_queue.cc:1035] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8591298b20482ca83420aab2e8c25b" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:52:51.906728 2704 consensus_queue.cc:1035] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "24c97562a33544a29a248018b453b8e5" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:52:51.919579 2710 mvcc.cc:205] Tried to move back new op lower bound from 7334341721711071232 to 7334341721511051264. Current Snapshot: MvccSnapshot[applied={T|T < 7334341721711071232}]
I20260928 15:52:51.923887 2711 mvcc.cc:205] Tried to move back new op lower bound from 7334341721711071232 to 7334341721511051264. Current Snapshot: MvccSnapshot[applied={T|T < 7334341721711071232}]
I20260928 15:52:51.937688 2182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:52:51.937949 2182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:52:51.942155 2182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e857370b1f1d42ba9d1ec6ec939746b6: Sending DeleteTablet for 3 replicas of tablet aa82b6cdb00945e2af1b4c3152cd92af
I20260928 15:52:51.943427 2476 tablet_service.cc:1514] Processing DeleteTablet for tablet aa82b6cdb00945e2af1b4c3152cd92af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:52:51 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:34606
I20260928 15:52:51.943634 2611 tablet_service.cc:1514] Processing DeleteTablet for tablet aa82b6cdb00945e2af1b4c3152cd92af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:52:51 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:49862
I20260928 15:52:51.944304 2332 tablet_service.cc:1514] Processing DeleteTablet for tablet aa82b6cdb00945e2af1b4c3152cd92af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:52:51 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:42912
I20260928 15:52:51.944780 2719 tablet_replica.cc:330] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: stopping tablet replica
I20260928 15:52:51.945353 2719 raft_consensus.cc:2229] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:52:51.945473 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 2263
I20260928 15:52:51.945966 2719 raft_consensus.cc:2258] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:52:51.947345 2721 tablet_replica.cc:330] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: stopping tablet replica
I20260928 15:52:51.948342 2721 raft_consensus.cc:2229] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:52:51.949270 2721 raft_consensus.cc:2258] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:52:51.949533 2719 ts_tablet_manager.cc:1905] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:52:51.952432 2720 tablet_replica.cc:330] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: stopping tablet replica
I20260928 15:52:51.952943 2720 raft_consensus.cc:2229] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:52:51.953488 2720 raft_consensus.cc:2258] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:52:51.956777 2721 ts_tablet_manager.cc:1905] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:52:51.957386 2719 ts_tablet_manager.cc:1918] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:52:51.957603 2719 log.cc:1196] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/aa82b6cdb00945e2af1b4c3152cd92af
I20260928 15:52:51.958163 2719 ts_tablet_manager.cc:1939] T aa82b6cdb00945e2af1b4c3152cd92af P 24c97562a33544a29a248018b453b8e5: Deleting consensus metadata
I20260928 15:52:51.960150 2168 catalog_manager.cc:4841] TS 24c97562a33544a29a248018b453b8e5 (127.22.157.194:37329): tablet aa82b6cdb00945e2af1b4c3152cd92af (table test-table [id=e9a273e26f7f4ba5b221f38ff66a5759]) successfully deleted
Sep 28 15:52:51 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:52:51.961700 2720 ts_tablet_manager.cc:1905] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:52:51.969112 2720 ts_tablet_manager.cc:1918] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:52:51.969349 2720 log.cc:1196] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/aa82b6cdb00945e2af1b4c3152cd92af
I20260928 15:52:51.970041 2720 ts_tablet_manager.cc:1939] T aa82b6cdb00945e2af1b4c3152cd92af P aa8591298b20482ca83420aab2e8c25b: Deleting consensus metadata
I20260928 15:52:51.972177 2167 catalog_manager.cc:4841] TS aa8591298b20482ca83420aab2e8c25b (127.22.157.195:43437): tablet aa82b6cdb00945e2af1b4c3152cd92af (table test-table [id=e9a273e26f7f4ba5b221f38ff66a5759]) successfully deleted
I20260928 15:52:51.972756 2721 ts_tablet_manager.cc:1918] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:52:51.972957 2721 log.cc:1196] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/aa82b6cdb00945e2af1b4c3152cd92af
I20260928 15:52:51.973165 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:51.947253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.22.157.193:42319)
0928 15:52:51.947424 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:51.947431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:51.950271 (+ 2840us) server_negotiation.cc:407] Connection header received
0928 15:52:51.950493 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:51.950505 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:51.950613 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:51.950722 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:51.952094 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.953477 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.955408 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:51.956112 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:51.956248 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:51.962162 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:51.962175 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:51.962188 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:51.962226 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:51.965801 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:51.966788 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:51.966796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:51.966798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:51.969706 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:51.970250 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:51.970256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:51.970260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:51.970818 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:51.971120 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:51.972194 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0928 15:52:51.972528 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 15:52:51.973757 2721 ts_tablet_manager.cc:1939] T aa82b6cdb00945e2af1b4c3152cd92af P c15b180b33d141a8a01b59d91b06f4db: Deleting consensus metadata
I20260928 15:52:51.974042 2392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:52:52.074743 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 2263
I20260928 15:52:52.074885 2167 catalog_manager.cc:4841] TS c15b180b33d141a8a01b59d91b06f4db (127.22.157.193:42319): tablet aa82b6cdb00945e2af1b4c3152cd92af (table test-table [id=e9a273e26f7f4ba5b221f38ff66a5759]) successfully deleted
I20260928 15:52:52.093663 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 2404
Sep 28 15:52:52 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:52:52.109787 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:52.094499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.22.157.194:37329)
0928 15:52:52.094674 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:52.094678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:52.094698 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:52:52.094791 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:52.094795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:52.094857 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:52.094937 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:52.096275 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:52.097150 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:52.098403 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:52.098827 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:52.098930 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:52.103583 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:52.103600 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:52.103616 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:52.103671 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:52.106958 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:52.107660 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:52.107668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:52.107670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:52.107803 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:52.108122 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:52.108125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:52.108127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:52.108487 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:52.108816 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:52.109147 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 15:52:52.109374 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:52:52.110486 2536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:52:52.203001 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 2404
I20260928 15:52:52.220991 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 2548
Sep 28 15:52:52 dist-test-slave-1vw0 krb5kdc[1701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:52:52.236862 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:52:52.221884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.22.157.195:43437)
0928 15:52:52.222062 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:52:52.222067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:52:52.222085 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:52:52.222196 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:52:52.222200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:52:52.222263 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:52:52.222403 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:52:52.223637 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:52.224339 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:52.225773 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:52:52.226163 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:52:52.226256 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:52:52.230765 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:52:52.230784 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:52:52.230801 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:52:52.230839 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:52:52.233792 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:52.234616 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:52.234625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:52.234627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:52.234763 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:52:52.235139 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:52:52.235144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:52:52.235147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:52:52.235525 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:52:52.235840 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 15:52:52.236131 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 15:52:52.236358 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:52:52.237568 2671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:52:52.330046 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 2548
I20260928 15:52:52.347247 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 2150
I20260928 15:52:52.348548 2213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:52:52.461007 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 2150
I20260928 15:52:52.482288 23159 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 15:52:52.818234 23159 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 15:52:52.818481 23159 mini_ranger.cc:66] Stopping Ranger...
I20260928 15:52:53.164486 23159 mini_ranger.cc:68] Stopped Ranger
2026-09-28 15:52:53.164 UTC [1731] LOG: received smart shutdown request
2026-09-28 15:52:53.169 UTC [1731] LOG: background worker "logical replication launcher" (PID 1740) exited with exit code 1
2026-09-28 15:52:53.169 UTC [1735] LOG: shutting down
2026-09-28 15:52:53.364 UTC [1731] LOG: database system is shut down
2026-09-28T15:52:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51550 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:52:53 dist-test-slave-1vw0 krb5kdc[2735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:52:53 dist-test-slave-1vw0 krb5kdc[2735](info): set up 2 sockets
Sep 28 15:52:53 dist-test-slave-1vw0 krb5kdc[2735](info): commencing operation
krb5kdc: starting...
W20260928 15:52:54.626027 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:52:54 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:52:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:52:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/rangeradmin_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/rangeradmin_127.22.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/rangerlookup_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/rangerlookup_127.22.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/HTTP_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/HTTP_127.22.157.212@KRBTEST.COM.keytab.
I20260928 15:52:54.983322 23159 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1s6B80/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/postgres -l logfile start
2026-09-28 15:52:57.668 UTC [2765] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 15:52:57.668 UTC [2765] LOG: listening on IPv4 address "127.22.157.212", port 44227
2026-09-28 15:52:57.672 UTC [2765] LOG: listening on Unix socket "/tmp/.s.PGSQL.44227"
2026-09-28 15:52:57.696 UTC [2768] LOG: database system was shut down at 2026-09-28 15:52:56 UTC
2026-09-28 15:52:57.700 UTC [2765] LOG: database system is ready to accept connections
I20260928 15:52:58.701287 23159 mini_postgres.cc:96] Postgres bound to 44227
2026-09-28 15:52:58.721 UTC [2776] FATAL: database "slave" does not exist
127.22.157.212:44227 - accepting connections
I20260928 15:52:58.721623 23159 mini_ranger.cc:161] Starting Ranger...
I20260928 15:52:58.752085 23159 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 15:52:59.248208 23159 mini_ranger.cc:87] Created ranger Postgres database
I20260928 15:52:59.248342 23159 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-admin
2026-09-28 15:52:59,556 [I] DB FLAVOR :POSTGRES
2026-09-28 15:52:59,557 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:52:59,557 [I] Checking connection..
2026-09-28 15:52:59,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 15:52:59,811 [I] Checking connection passed.
2026-09-28 15:52:59,811 [I] --------- Verifying version history table ---------
2026-09-28 15:52:59,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:53:00,059 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:53:00,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:53:00,316 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 15:53:00,316 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 15:53:00,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 15:53:00.589 UTC [2840] WARNING: there is no transaction in progress
2026-09-28 15:53:00,596 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 15:53:00,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 15:53:00,853 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 15:53:00,854 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:53:00,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 15:53:01,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 15:53:01,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1vw0.c.gcp-upstream.internal','N') ;"
2026-09-28 15:53:01,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:53:01,867 [I] Table x_portal_user does not exist in database ranger
2026-09-28 15:53:01,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:53:02,126 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 15:53:02,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:53:02,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:53:02,636 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 15:53:02,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 15:53:03.646 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.664 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.681 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.698 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.714 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.736 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.807 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.815 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.832 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.843 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.854 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.869 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.880 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.887 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.894 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:03.901 UTC [2950] WARNING: there is no transaction in progress
2026-09-28 15:53:04,581 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 15:53:04,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 15:53:04,809 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 15:53:04,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 15:53:05,069 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 15:53:05,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 15:53:05,304 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 15:53:05,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 15:53:05,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 15:53:05,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1vw0.c.gcp-upstream.internal';"
2026-09-28 15:53:06,023 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 15:53:06,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 15:53:06,281 [I] DB_PATCHES have already been applied
I20260928 15:53:06.287478 23159 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-admin:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1s6B80/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 15:53:06.287568 23159 mini_ranger.cc:193] Using host: 127.22.157.212
I20260928 15:53:06.304330 23159 mini_ranger.cc:227] Ranger admin URL: http://127.22.157.212:37985
Sep 28, 2026 3:53:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:53:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:53:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 3:53:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 3:53:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 3:53:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 3:53:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 15:53:07.458463 23159 mini_ranger.cc:160] Time spent starting Ranger: real 8.737s user 0.004s sys 0.045s
Sep 28, 2026 3:53:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 3:53:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 3:53:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 3:53:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 3:53:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 15:53:29.309044 23159 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/rangerkms_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/rangerkms_127.22.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.157.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.157.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.157.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/HTTP_127.22.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.157.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/HTTP_127.22.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 15:53:29 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610809, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 15:53:29.501523 23159 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 15:53:29.524901 23159 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 15:53:30.496608 23159 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 15:53:30.496791 23159 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-kms
I20260928 15:53:30.496855 23159 mini_ranger_kms.cc:227] Using postgres at 127.22.157.212:44227
2026-09-28 15:53:30,867 [I] DB FLAVOR :POSTGRES
2026-09-28 15:53:30,867 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 15:53:30,867 [I] Checking connection
2026-09-28 15:53:30,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 15:53:31,105 [I] connection success
2026-09-28 15:53:31,106 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 15:53:31,106 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 15:53:31,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 15:53:31,355 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 15:53:31,356 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 15:53:31,356 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 15:53:31,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.157.212:44227/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 15:53:31,635 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 15:53:32.164093 23159 mini_ranger_kms.cc:310] Created kms service
I20260928 15:53:32.374704 23159 mini_ranger_kms.cc:326] Created kudu user
I20260928 15:53:32.423491 23159 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 15:53:32 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610809, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:32.858364 23159 mini_ranger_kms.cc:384] Added ranger policy
I20260928 15:53:32.858704 23159 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-kms:/tmp/dist-test-task1s6B80/build/asan/bin/postgresql.jar:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1s6B80/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 15:53:32.858764 23159 mini_ranger_kms.cc:240] Using host: 127.22.157.212
I20260928 15:53:32.878924 23159 mini_ranger_kms.cc:276] Ranger KMS PID: 3132
I20260928 15:53:32.879081 23159 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.157.212:58025
Sep 28, 2026 3:53:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 3:53:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 3:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 3:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 3:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 3:53:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 15:53:35 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610815, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.157.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:35.243726 23159 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.742s user 0.097s sys 0.296s
I20260928 15:53:35.244010 23159 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 15:53:35.244203 23159 mini_ranger_kms.cc:401] 127.22.157.212:58025/kms/v1/keys
Sep 28 15:53:35 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610815, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.157.212@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
Sep 28 15:53:35 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610815, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.157.212@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
Sep 28 15:53:38 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:53:38.457635 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42647
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:42647
--ranger_config_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:58025/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:53:38.818774 3184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:38.819113 3184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:38.819303 3184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:38.827026 3184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:53:38.827185 3184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:38.827289 3184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:38.827366 3184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:53:38.827437 3184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:53:38.838236 3184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.157.212:58025/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:42647
--ranger_config_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:38.839604 3184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:38.841161 3184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:38.854566 3189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:38.854646 3192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:38.854624 3190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:38.855194 3184 server_base.cc:1034] running on GCE node
I20260928 15:53:38.856089 3184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:38.857636 3184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:38.858848 3184 hybrid_clock.cc:648] HybridClock initialized: now 1790610818858798 us; error 48 us; skew 500 ppm
I20260928 15:53:38.859223 3184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:38 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610818, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:38.862463 3184 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:53:38.864058 3184 webserver.cc:460] Webserver started at http://127.22.157.254:41353/ using document root <none> and password file <none>
I20260928 15:53:38.864869 3184 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:38.865010 3184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:38.865383 3184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:53:38 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610818, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:38.930015 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "a732e0db6809427b9939c5adc7a27199"
format_stamp: "Formatted at 2026-09-28 15:53:38 on dist-test-slave-1vw0"
server_key: "db33d56ec9ac6fdace76384594f6b1ad"
server_key_iv: "4fa2dfe6841073f2b4ea6587164db833"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:38.930918 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "a732e0db6809427b9939c5adc7a27199"
format_stamp: "Formatted at 2026-09-28 15:53:38 on dist-test-slave-1vw0"
server_key: "db33d56ec9ac6fdace76384594f6b1ad"
server_key_iv: "4fa2dfe6841073f2b4ea6587164db833"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:38.936900 3184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:53:38.956498 3199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:38.958187 3184 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:53:38.958384 3184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "a732e0db6809427b9939c5adc7a27199"
format_stamp: "Formatted at 2026-09-28 15:53:38 on dist-test-slave-1vw0"
server_key: "db33d56ec9ac6fdace76384594f6b1ad"
server_key_iv: "4fa2dfe6841073f2b4ea6587164db833"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:38.958632 3184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:38.979220 3184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:38.983604 3184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:38.983960 3184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:39.004274 3184 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:42647
I20260928 15:53:39.004282 3250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:42647 every 8 connection(s)
I20260928 15:53:39.005888 3184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:53:39.007597 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 3184
I20260928 15:53:39.007814 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:53:39.008671 23159 external_mini_cluster.cc:1428] Setting key f119ff44e38645f0e45c126fbedc9b87
I20260928 15:53:39.011528 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:39.029567 3251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Bootstrap starting.
I20260928 15:53:39.033608 3251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:39.034973 3251 log.cc:824] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:39.039870 3251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: No bootstrap required, opened a new log
Sep 28 15:53:39 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:39.050482 3251 raft_consensus.cc:348] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } }
I20260928 15:53:39.051133 3251 raft_consensus.cc:374] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:39.051357 3251 raft_consensus.cc:729] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a732e0db6809427b9939c5adc7a27199, State: Initialized, Role: FOLLOWER
I20260928 15:53:39.052089 3251 consensus_queue.cc:260] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } }
I20260928 15:53:39.052382 3251 raft_consensus.cc:388] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:53:39.052525 3251 raft_consensus.cc:482] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:53:39.052752 3251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:39.053040 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:39.010957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.22.157.254:42647)
0928 15:53:39.026435 (+ 15478us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:39.026457 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:39.026517 (+ 60us) server_negotiation.cc:407] Connection header received
0928 15:53:39.029196 (+ 2679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:39.029231 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:39.029841 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:39.030339 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:39.032223 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:39.034428 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:39.036843 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:39.038018 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:39.038209 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:39.043346 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:39.043359 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:39.043382 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:39.043442 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:39.047784 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:39.048710 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:39.048746 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:39.048756 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:39.048946 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:39.049417 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:39.049422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:39.049425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:39.050185 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:39.050474 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:39.050998 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 15:53:39.051498 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15211,"thread_start_us":14957,"threads_started":1}
I20260928 15:53:39.055732 3251 raft_consensus.cc:504] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } }
I20260928 15:53:39.056317 3251 leader_election.cc:302] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a732e0db6809427b9939c5adc7a27199; no voters:
I20260928 15:53:39.057155 3251 leader_election.cc:288] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:53:39.057273 3256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:53:39.059810 3251 sys_catalog.cc:564] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:53:39.061067 3256 raft_consensus.cc:686] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 1 LEADER]: Becoming Leader. State: Replica: a732e0db6809427b9939c5adc7a27199, State: Running, Role: LEADER
I20260928 15:53:39.061865 3256 consensus_queue.cc:237] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } }
I20260928 15:53:39.065466 3257 sys_catalog.cc:455] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a732e0db6809427b9939c5adc7a27199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } } }
I20260928 15:53:39.066094 3257 sys_catalog.cc:458] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:39.065511 3258 sys_catalog.cc:455] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a732e0db6809427b9939c5adc7a27199. Latest consensus state: current_term: 1 leader_uuid: "a732e0db6809427b9939c5adc7a27199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } } }
I20260928 15:53:39.068203 3258 sys_catalog.cc:458] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:39.089376 3264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:53:39.095045 3264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:53:39.095903 3251 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:53:39.119762 3264 catalog_manager.cc:1347] Generated new cluster ID: fe95dd139d8d42dca5bcd9aee3e404d9
I20260928 15:53:39.120000 3264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:53:39.156262 3264 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:53:39.158592 3264 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:53:39.168859 3264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Generated new TSK 0
I20260928 15:53:39.170176 3264 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:53:39.929642 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:58025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42647
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:53:40 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610820, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 15:53:40.373780 3296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:40.374279 3296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:40.374454 3296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:40.386528 3296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:40.386695 3296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:40.386904 3296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:53:40.403508 3296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.157.212:58025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:40.405594 3296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:40.407426 3296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:40.424147 3309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:40.424045 3307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:40.425180 3306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:40.425689 3296 server_base.cc:1034] running on GCE node
I20260928 15:53:40.426447 3296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:40.428596 3296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:40.429949 3296 hybrid_clock.cc:648] HybridClock initialized: now 1790610820429837 us; error 97 us; skew 500 ppm
I20260928 15:53:40.430393 3296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:40 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610820, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:40.434944 3296 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:53:40.449260 3296 webserver.cc:460] Webserver started at http://127.22.157.193:41673/ using document root <none> and password file <none>
I20260928 15:53:40.450367 3296 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:40.450531 3296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:40.450937 3296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:53:40 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610820, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:40.479333 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "3128ea3052a74d40b9d9ff741c19bb29"
format_stamp: "Formatted at 2026-09-28 15:53:40 on dist-test-slave-1vw0"
server_key: "4b141f98960040f7564f503f85e82eac"
server_key_iv: "9f79f838b5024d61fc55a9b2898b5167"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:40.482028 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "3128ea3052a74d40b9d9ff741c19bb29"
format_stamp: "Formatted at 2026-09-28 15:53:40 on dist-test-slave-1vw0"
server_key: "4b141f98960040f7564f503f85e82eac"
server_key_iv: "9f79f838b5024d61fc55a9b2898b5167"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:40.495898 3296 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.004s
I20260928 15:53:40.526840 3316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:40.529021 3296 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.007s
I20260928 15:53:40.529316 3296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "3128ea3052a74d40b9d9ff741c19bb29"
format_stamp: "Formatted at 2026-09-28 15:53:40 on dist-test-slave-1vw0"
server_key: "4b141f98960040f7564f503f85e82eac"
server_key_iv: "9f79f838b5024d61fc55a9b2898b5167"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:40.529670 3296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:40.609743 3296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:40.615835 3296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:40.616299 3296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:40.618826 3296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:53:40.622313 3296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:53:40.622504 3296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 15:53:40.622735 3296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:53:40.622869 3296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:40.699828 3296 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:35673
I20260928 15:53:40.699874 3429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:35673 every 8 connection(s)
I20260928 15:53:40.701596 3296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:53:40.711730 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 3296
I20260928 15:53:40.711968 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:53:40.712894 23159 external_mini_cluster.cc:1428] Setting key 613e35b2bc2a6add7c657a15afc20486
Sep 28 15:53:40 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610820, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:40.751070 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:40.705399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:42647 (local address 127.22.157.193:39825)
0928 15:53:40.710396 (+ 4997us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:40.710467 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:40.713019 (+ 2552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:40.722657 (+ 9638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:40.722673 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:40.723450 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:40.724446 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:40.724476 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:40.726381 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:40.726388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:40.728667 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:40.728685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:40.734932 (+ 6247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:40.734942 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:40.735334 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:40.736446 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:40.736481 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:40.739495 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:40.743958 (+ 4463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:40.743970 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:40.743991 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:40.744647 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:40.747079 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:40.747085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:40.747089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:40.747390 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:40.748102 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:40.748112 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:40.748598 (+ 486us) client_negotiation.cc:769] Sending connection context
0928 15:53:40.748944 (+ 346us) client_negotiation.cc:240] Negotiation successful
0928 15:53:40.749557 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4548,"thread_start_us":8820,"threads_started":1}
I20260928 15:53:40.751363 3432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:40.704784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:39825 (local address 127.22.157.254:42647)
0928 15:53:40.721886 (+ 17102us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:40.721894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:40.721922 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:53:40.722003 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:40.722007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:40.722100 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:40.722244 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:40.724863 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:40.726146 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:40.733839 (+ 7693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:40.734583 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:40.734749 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:40.740019 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:40.740027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:40.740033 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:40.740112 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:40.743768 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:40.746589 (+ 2821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:40.746596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:40.746600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:40.746773 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:40.747543 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:40.747549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:40.747552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:40.747936 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:40.748092 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:40.750751 (+ 2659us) server_negotiation.cc:299] Negotiation successful
0928 15:53:40.750992 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16926,"thread_start_us":17492,"threads_started":1}
I20260928 15:53:40.752899 3430 heartbeater.cc:347] Connected to a master server at 127.22.157.254:42647
I20260928 15:53:40.753242 3430 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:40.754113 3430 heartbeater.cc:511] Master 127.22.157.254:42647 requested a full tablet report, sending...
I20260928 15:53:40.756635 3216 ts_manager.cc:194] Registered new tserver with Master: 3128ea3052a74d40b9d9ff741c19bb29 (127.22.157.193:35673)
I20260928 15:53:40.759377 3216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:39825
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Sep 28 15:53:40 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610820, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:53:40.808525 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:58025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42647
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:53:41 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610820, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
W20260928 15:53:41.315762 3436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:41.316308 3436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:41.316514 3436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:41.328564 3436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:41.328853 3436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:41.329114 3436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:53:41.345906 3436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.157.212:58025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:41.347781 3436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:41.349664 3436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:41.367368 3443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:41.367475 3444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:41.372795 3446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:41.373351 3436 server_base.cc:1034] running on GCE node
I20260928 15:53:41.374378 3436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:41.375861 3436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:41.377167 3436 hybrid_clock.cc:648] HybridClock initialized: now 1790610821377102 us; error 59 us; skew 500 ppm
I20260928 15:53:41.377600 3436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:41 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:41.382017 3436 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:53:41.386981 3436 webserver.cc:460] Webserver started at http://127.22.157.194:35799/ using document root <none> and password file <none>
I20260928 15:53:41.388089 3436 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:41.388293 3436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:41.388772 3436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:53:41 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:41.411912 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "4db47406d5f946bdb454ee797d48dabb"
format_stamp: "Formatted at 2026-09-28 15:53:41 on dist-test-slave-1vw0"
server_key: "4945a718729065a064b6c43dbcf7af3b"
server_key_iv: "abd8290f496cf48ee9d547b9f0f61e0e"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:41.413266 3436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "4db47406d5f946bdb454ee797d48dabb"
format_stamp: "Formatted at 2026-09-28 15:53:41 on dist-test-slave-1vw0"
server_key: "4945a718729065a064b6c43dbcf7af3b"
server_key_iv: "abd8290f496cf48ee9d547b9f0f61e0e"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:41.420459 3436 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260928 15:53:41.439203 3461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:41.440974 3436 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:53:41.441164 3436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "4db47406d5f946bdb454ee797d48dabb"
format_stamp: "Formatted at 2026-09-28 15:53:41 on dist-test-slave-1vw0"
server_key: "4945a718729065a064b6c43dbcf7af3b"
server_key_iv: "abd8290f496cf48ee9d547b9f0f61e0e"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:41.441386 3436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:41.476977 3436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:41.481307 3436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:41.481642 3436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:41.482751 3436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:53:41.484336 3436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:53:41.484411 3436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:41.484566 3436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:53:41.484658 3436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:41.520341 3436 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:34463
I20260928 15:53:41.520591 3573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:34463 every 8 connection(s)
I20260928 15:53:41.522002 3436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:53:41.528231 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 3436
I20260928 15:53:41.528438 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:53:41.529053 23159 external_mini_cluster.cc:1428] Setting key 636f8d3258ba4f8a4e9cee1796dd8511
Sep 28 15:53:41 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:41.546041 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:41.524703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:49743 (local address 127.22.157.254:42647)
0928 15:53:41.525231 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:41.525236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:41.526209 (+ 973us) server_negotiation.cc:407] Connection header received
0928 15:53:41.528123 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:41.528127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:41.528208 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:41.528308 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:41.530207 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:41.530996 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:41.533607 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:41.534029 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:41.534134 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:41.538874 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:41.538884 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:41.538892 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:41.538945 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:41.542344 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:41.543394 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:41.543398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:41.543401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:41.543527 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:41.544061 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:41.544064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:41.544067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:41.544361 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:41.544474 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:41.545494 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0928 15:53:41.545704 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":254,"threads_started":1}
I20260928 15:53:41.547623 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:41.525307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:42647 (local address 127.22.157.194:49743)
0928 15:53:41.526024 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:41.526063 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:41.527760 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:41.528620 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:41.528634 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:41.529282 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:41.530023 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:41.530042 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:41.531167 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:41.531171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:41.533417 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:41.533449 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:41.534284 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:41.534294 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:41.534634 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:41.535428 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:41.535547 (+ 119us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:41.538564 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:41.542509 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:41.542522 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:41.542546 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:41.543253 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:41.543632 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:41.543639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:41.543646 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:41.543939 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:41.544497 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:41.544509 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:41.545031 (+ 522us) client_negotiation.cc:769] Sending connection context
0928 15:53:41.545354 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 15:53:41.546011 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":285,"threads_started":1}
I20260928 15:53:41.549300 3574 heartbeater.cc:347] Connected to a master server at 127.22.157.254:42647
I20260928 15:53:41.549661 3574 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:41.550426 3574 heartbeater.cc:511] Master 127.22.157.254:42647 requested a full tablet report, sending...
I20260928 15:53:41.552012 3216 ts_manager.cc:194] Registered new tserver with Master: 4db47406d5f946bdb454ee797d48dabb (127.22.157.194:34463)
I20260928 15:53:41.552951 3216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:49743
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:53:41.594254 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:58025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42647
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:53:41.763465 3430 heartbeater.cc:503] Master 127.22.157.254:42647 was elected leader, sending a full tablet report...
W20260928 15:53:41.941537 3580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:41.941844 3580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:41.942014 3580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:41.949877 3580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:41.949996 3580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:41.950174 3580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:53:41.960667 3580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.157.212:58025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:42647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:41.962294 3580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:41.963723 3580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:41.977795 3585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:41.977852 3588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:41.977819 3586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:41.979056 3580 server_base.cc:1034] running on GCE node
I20260928 15:53:41.979663 3580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:41.980646 3580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:41.981855 3580 hybrid_clock.cc:648] HybridClock initialized: now 1790610821981822 us; error 26 us; skew 500 ppm
I20260928 15:53:41.982203 3580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:41 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:41.990829 3580 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:53:41.992645 3580 webserver.cc:460] Webserver started at http://127.22.157.195:40011/ using document root <none> and password file <none>
I20260928 15:53:41.993436 3580 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:41.993575 3580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:41.993912 3580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 15:53:41 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:42.011107 3580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "a11921c6a91e4bbebf605354f19c53de"
format_stamp: "Formatted at 2026-09-28 15:53:42 on dist-test-slave-1vw0"
server_key: "c90565487cacbb235005dba6a73edc4b"
server_key_iv: "a0eb9c68c032301c3f20bcc579d05984"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:42.012189 3580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "a11921c6a91e4bbebf605354f19c53de"
format_stamp: "Formatted at 2026-09-28 15:53:42 on dist-test-slave-1vw0"
server_key: "c90565487cacbb235005dba6a73edc4b"
server_key_iv: "a0eb9c68c032301c3f20bcc579d05984"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:42.018522 3580 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 15:53:42.035936 3595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:42.037755 3580 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:53:42.037978 3580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "a11921c6a91e4bbebf605354f19c53de"
format_stamp: "Formatted at 2026-09-28 15:53:42 on dist-test-slave-1vw0"
server_key: "c90565487cacbb235005dba6a73edc4b"
server_key_iv: "a0eb9c68c032301c3f20bcc579d05984"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:42.038230 3580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:42.055173 3580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:42.059561 3580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:42.059916 3580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:42.061041 3580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:53:42.062573 3580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:53:42.062682 3580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:42.062835 3580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:53:42.062906 3580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:42.098176 3580 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:35255
I20260928 15:53:42.098189 3709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:35255 every 8 connection(s)
I20260928 15:53:42.099632 3580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:53:42.102038 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 3580
I20260928 15:53:42.102222 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:53:42.102761 23159 external_mini_cluster.cc:1428] Setting key e32f4f62568691097a2ff18c8d14f661
Sep 28 15:53:42 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610821, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:42.122177 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:42.102705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:59493 (local address 127.22.157.254:42647)
0928 15:53:42.103338 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:42.103346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:42.104783 (+ 1437us) server_negotiation.cc:407] Connection header received
0928 15:53:42.107018 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:42.107022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:42.107096 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:42.107201 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:42.108877 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.109651 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.111392 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.111882 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.111982 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:42.115723 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:42.115729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:42.115732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:42.115768 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:42.118993 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:42.119759 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:42.119765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:42.119769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:42.119901 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:42.120356 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:42.120361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:42.120366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:42.120645 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:42.120833 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:42.121562 (+ 729us) server_negotiation.cc:299] Negotiation successful
0928 15:53:42.121809 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":295,"threads_started":1}
I20260928 15:53:42.123252 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:42.103396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:42647 (local address 127.22.157.195:59493)
0928 15:53:42.104389 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:42.104451 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:42.106746 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:42.107472 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:42.107482 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:42.107987 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:42.108655 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:42.108673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.109837 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.109842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:42.111253 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:42.111266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.112114 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.112119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:42.112338 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:42.113154 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:42.113179 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:42.115487 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:42.119153 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:42.119166 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:42.119181 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:42.119633 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:42.120014 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:42.120018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:42.120022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:42.120235 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:42.120816 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:42.120823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:42.121184 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 15:53:42.121458 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 15:53:42.122017 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":468,"threads_started":1}
I20260928 15:53:42.124753 3710 heartbeater.cc:347] Connected to a master server at 127.22.157.254:42647
I20260928 15:53:42.125043 3710 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:42.125712 3710 heartbeater.cc:511] Master 127.22.157.254:42647 requested a full tablet report, sending...
I20260928 15:53:42.127168 3216 ts_manager.cc:194] Registered new tserver with Master: a11921c6a91e4bbebf605354f19c53de (127.22.157.195:35255)
I20260928 15:53:42.127976 3216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:59493
I20260928 15:53:42.131693 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:53:42.133425 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 3296
Sep 28 15:53:42 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:53:42.151235 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:42.134509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.22.157.193:35673)
0928 15:53:42.135349 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:42.135358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:42.135389 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:53:42.135460 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:42.135466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:42.135670 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:42.135849 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:42.137498 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.138530 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.140066 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.140647 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.140843 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:42.145523 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:42.145538 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:42.145553 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:42.145594 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:42.148531 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:42.149305 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:42.149312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:42.149315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:42.149466 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:42.149838 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:42.149842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:42.149844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:42.150219 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:42.150454 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:42.150705 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 15:53:42.150913 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":474,"threads_started":1}
I20260928 15:53:42.152658 3425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:42.236416 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 3296
I20260928 15:53:42.252887 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 3436
Sep 28 15:53:42 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:53:42.271142 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:42.253859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.22.157.194:34463)
0928 15:53:42.254484 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:42.254493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:42.254520 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:53:42.254588 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:42.254594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:42.254813 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:42.254943 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:42.256247 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.257458 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.259082 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.259650 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.259829 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:42.265067 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:42.265080 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:42.265096 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:42.265139 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:42.268090 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:42.268880 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:42.268888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:42.268890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:42.269026 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:42.269439 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:42.269445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:42.269448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:42.269823 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:42.270158 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:42.270468 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 15:53:42.270754 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":352,"threads_started":1}
I20260928 15:53:42.272491 3569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:42.355644 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 3436
I20260928 15:53:42.376603 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 3580
Sep 28 15:53:42 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:53:42.393611 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:42.377740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.22.157.195:35255)
0928 15:53:42.378389 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:42.378396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:42.378423 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:53:42.378484 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:42.378491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:42.378710 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:42.378833 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:42.380078 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.381261 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.382552 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:42.383143 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:42.383241 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:42.387885 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:42.387897 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:42.387912 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:42.387951 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:42.390953 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:42.391648 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:42.391655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:42.391657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:42.391783 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:42.392117 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:42.392120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:42.392122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:42.392454 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:42.392693 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:42.393057 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 15:53:42.393284 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":326,"threads_started":1}
I20260928 15:53:42.394976 3705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:42.481608 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 3580
I20260928 15:53:42.498277 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 3184
I20260928 15:53:42.499924 3246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:42.611793 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 3184
I20260928 15:53:42.638218 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42647
--webserver_interface=127.22.157.254
--webserver_port=41353
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:42647
--ranger_config_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:58025/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.157.212:58025/kms with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:53:42.998975 3722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:42.999300 3722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:42.999469 3722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:43.007438 3722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:53:43.007586 3722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:43.007718 3722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:43.007834 3722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:53:43.007937 3722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:53:43.018946 3722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.157.212:58025/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:42647
--ranger_config_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=41353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:43.020419 3722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:43.021929 3722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:43.035156 3728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:43.035156 3730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:43.035192 3727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:43.036113 3722 server_base.cc:1034] running on GCE node
I20260928 15:53:43.037075 3722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:43.038592 3722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:43.039846 3722 hybrid_clock.cc:648] HybridClock initialized: now 1790610823039809 us; error 57 us; skew 500 ppm
I20260928 15:53:43.040256 3722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:43 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610823, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:43.043756 3722 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:53:43.045509 3722 webserver.cc:460] Webserver started at http://127.22.157.254:41353/ using document root <none> and password file <none>
I20260928 15:53:43.046195 3722 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:43.046347 3722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:53:43 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610823, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:43.121685 3722 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:53:43.125144 3739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:43.126719 3722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:53:43.126962 3722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "a732e0db6809427b9939c5adc7a27199"
format_stamp: "Formatted at 2026-09-28 15:53:38 on dist-test-slave-1vw0"
server_key: "db33d56ec9ac6fdace76384594f6b1ad"
server_key_iv: "4fa2dfe6841073f2b4ea6587164db833"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:43.128053 3722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:43.186602 3722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:43.191179 3722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:43.191521 3722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:43.211709 3722 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:42647
I20260928 15:53:43.211752 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:42647 every 8 connection(s)
I20260928 15:53:43.213351 3722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:53:43.218657 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 3722
I20260928 15:53:43.219807 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:58025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:35673
--local_ip_for_outbound_sockets=127.22.157.193
--tserver_master_addrs=127.22.157.254:42647
--webserver_port=41673
--webserver_interface=127.22.157.193
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.157.212:58025/kms with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:53:43.222268 3791 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:53:43.228336 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Bootstrap starting.
I20260928 15:53:43.250034 3791 log.cc:824] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:43.259670 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:53:43.260304 3791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Bootstrap complete.
I20260928 15:53:43.270459 3791 raft_consensus.cc:348] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } }
I20260928 15:53:43.271431 3791 raft_consensus.cc:729] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a732e0db6809427b9939c5adc7a27199, State: Initialized, Role: FOLLOWER
I20260928 15:53:43.272153 3791 consensus_queue.cc:260] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } }
I20260928 15:53:43.272394 3791 raft_consensus.cc:388] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:53:43.272523 3791 raft_consensus.cc:482] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:53:43.272742 3791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:53:43.277268 3791 raft_consensus.cc:504] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } }
I20260928 15:53:43.277830 3791 leader_election.cc:302] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a732e0db6809427b9939c5adc7a27199; no voters:
I20260928 15:53:43.278579 3791 leader_election.cc:288] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:53:43.278748 3795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:53:43.279902 3795 raft_consensus.cc:686] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [term 2 LEADER]: Becoming Leader. State: Replica: a732e0db6809427b9939c5adc7a27199, State: Running, Role: LEADER
I20260928 15:53:43.280694 3795 consensus_queue.cc:237] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } }
I20260928 15:53:43.281076 3791 sys_catalog.cc:564] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:53:43.283432 3797 sys_catalog.cc:455] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a732e0db6809427b9939c5adc7a27199. Latest consensus state: current_term: 2 leader_uuid: "a732e0db6809427b9939c5adc7a27199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } } }
I20260928 15:53:43.283811 3797 sys_catalog.cc:458] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:43.283527 3796 sys_catalog.cc:455] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a732e0db6809427b9939c5adc7a27199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a732e0db6809427b9939c5adc7a27199" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42647 } } }
I20260928 15:53:43.283979 3796 sys_catalog.cc:458] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199 [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:43.294320 3801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:53:43.301927 3801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:53:43.302795 3791 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 15:53:43.303609 3801 catalog_manager.cc:1259] Loaded cluster ID: fe95dd139d8d42dca5bcd9aee3e404d9
I20260928 15:53:43.303706 3801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:53:43.308537 3801 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:53:43.319610 3801 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Loaded TSK: 0
I20260928 15:53:43.322046 3801 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:53:43.662506 3793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:43.662961 3793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:43.663134 3793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:43.673725 3793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:43.673885 3793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:43.674095 3793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:53:43.687942 3793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.157.212:58025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:35673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=41673
--tserver_master_addrs=127.22.157.254:42647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:43.689520 3793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:43.691386 3793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:43.724146 3829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:43.729176 3830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:43.731030 3832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:43.731967 3793 server_base.cc:1034] running on GCE node
I20260928 15:53:43.733006 3793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:43.734513 3793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:43.735808 3793 hybrid_clock.cc:648] HybridClock initialized: now 1790610823735717 us; error 91 us; skew 500 ppm
I20260928 15:53:43.736219 3793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:43 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610823, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:43.740387 3793 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:53:43.761302 3793 webserver.cc:460] Webserver started at http://127.22.157.193:41673/ using document root <none> and password file <none>
I20260928 15:53:43.762543 3793 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:43.762755 3793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:53:43 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610823, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:43.817651 3793 fs_manager.cc:709] Time spent opening directory manager: real 0.023s user 0.009s sys 0.012s
I20260928 15:53:43.826110 3840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:43.830777 3793 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.000s
I20260928 15:53:43.833813 3793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "3128ea3052a74d40b9d9ff741c19bb29"
format_stamp: "Formatted at 2026-09-28 15:53:40 on dist-test-slave-1vw0"
server_key: "4b141f98960040f7564f503f85e82eac"
server_key_iv: "9f79f838b5024d61fc55a9b2898b5167"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:43.838567 3793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:43.906798 3793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:43.913841 3793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:43.914294 3793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:43.917937 3793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:53:43.920006 3793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:53:43.920149 3793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:43.920275 3793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:53:43.920357 3793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:43.996846 3793 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:35673
I20260928 15:53:43.996961 3954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:35673 every 8 connection(s)
I20260928 15:53:43.998863 3793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:53:44.005368 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 3793
I20260928 15:53:44.006255 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:58025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:34463
--local_ip_for_outbound_sockets=127.22.157.194
--tserver_master_addrs=127.22.157.254:42647
--webserver_port=35799
--webserver_interface=127.22.157.194
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.157.212:58025/kms with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:53:44 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610823, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:44.035620 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:44.002336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:42647 (local address 127.22.157.193:57215)
0928 15:53:44.003803 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:44.005033 (+ 1230us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:44.008114 (+ 3081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:44.010393 (+ 2279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:44.010405 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:44.011070 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:44.011926 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:44.011955 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:44.014570 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:44.014579 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:44.016874 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:44.016891 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:44.019098 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:44.019111 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:44.019485 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:44.020435 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:44.020475 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:44.024063 (+ 3588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:44.029209 (+ 5146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:44.029221 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:44.029239 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:44.029947 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:44.030539 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:44.030557 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:44.030561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:44.031187 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:44.032497 (+ 1310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:44.032505 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:44.033077 (+ 572us) client_negotiation.cc:769] Sending connection context
0928 15:53:44.033434 (+ 357us) client_negotiation.cc:240] Negotiation successful
0928 15:53:44.034078 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":793,"threads_started":1}
I20260928 15:53:44.037140 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:44.002334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:57215 (local address 127.22.157.254:42647)
0928 15:53:44.003189 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:44.003209 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:44.005209 (+ 2000us) server_negotiation.cc:407] Connection header received
0928 15:53:44.009119 (+ 3910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:44.009141 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:44.009710 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:44.010161 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:44.012214 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:44.014324 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:44.017069 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:44.018262 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:44.018634 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:44.024538 (+ 5904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:44.024568 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:44.024598 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:44.024657 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:44.028929 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:44.030153 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:44.030163 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:44.030172 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:44.030376 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:44.031332 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:44.031337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:44.031341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:44.032185 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:44.032404 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:44.035040 (+ 2636us) server_negotiation.cc:299] Negotiation successful
0928 15:53:44.035588 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":310,"threads_started":1}
I20260928 15:53:44.040900 3955 heartbeater.cc:347] Connected to a master server at 127.22.157.254:42647
I20260928 15:53:44.041256 3955 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:44.042161 3955 heartbeater.cc:511] Master 127.22.157.254:42647 requested a full tablet report, sending...
W20260928 15:53:44.045744 3955 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:42647 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:53:44.045982 3955 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:44.046288 3955 heartbeater.cc:511] Master 127.22.157.254:42647 requested a full tablet report, sending...
I20260928 15:53:44.047801 3955 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:44.048171 3955 heartbeater.cc:511] Master 127.22.157.254:42647 requested a full tablet report, sending...
W20260928 15:53:44.049405 3955 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:53:44.401727 3959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:44.402063 3959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:44.402244 3959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:44.411165 3959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:44.411325 3959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:44.411530 3959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:53:44.422755 3959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.157.212:58025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:34463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=35799
--tserver_master_addrs=127.22.157.254:42647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:44.424185 3959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:44.425645 3959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:44.442446 3972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:44.442487 3971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:44.444566 3974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:44.445645 3959 server_base.cc:1034] running on GCE node
I20260928 15:53:44.446548 3959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:44.448697 3959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:44.450116 3959 hybrid_clock.cc:648] HybridClock initialized: now 1790610824450017 us; error 69 us; skew 500 ppm
I20260928 15:53:44.450616 3959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:44 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610824, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:44.455199 3959 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:53:44.462615 3959 webserver.cc:460] Webserver started at http://127.22.157.194:35799/ using document root <none> and password file <none>
I20260928 15:53:44.463459 3959 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:44.463646 3959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:53:44 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610824, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:44.502815 3959 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.007s
I20260928 15:53:44.506522 3984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:44.508363 3959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 15:53:44.508554 3959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "4db47406d5f946bdb454ee797d48dabb"
format_stamp: "Formatted at 2026-09-28 15:53:41 on dist-test-slave-1vw0"
server_key: "4945a718729065a064b6c43dbcf7af3b"
server_key_iv: "abd8290f496cf48ee9d547b9f0f61e0e"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:44.510088 3959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:44.546507 3959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:44.552099 3959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:44.552488 3959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:44.555230 3959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:53:44.557256 3959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:53:44.557389 3959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:44.557531 3959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:53:44.557622 3959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:44.618984 3959 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:34463
I20260928 15:53:44.619069 4099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:34463 every 8 connection(s)
I20260928 15:53:44.621037 3959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:53:44.621932 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 3959
I20260928 15:53:44.622737 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.157.212:58025/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:35255
--local_ip_for_outbound_sockets=127.22.157.195
--tserver_master_addrs=127.22.157.254:42647
--webserver_port=40011
--webserver_interface=127.22.157.195
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.157.212:58025/kms with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:53:44 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610824, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:44.658310 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:44.631576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:38923 (local address 127.22.157.254:42647)
0928 15:53:44.633007 (+ 1431us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:44.633013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:44.636776 (+ 3763us) server_negotiation.cc:407] Connection header received
0928 15:53:44.637884 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:44.637890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:44.637987 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:44.638096 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:44.640679 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:44.641951 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:44.644391 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:44.645120 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:44.645234 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:44.650227 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:44.650234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:44.650239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:44.650283 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:44.653972 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:44.655104 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:44.655110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:44.655113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:44.655267 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:44.655909 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:44.655913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:44.655916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:44.656321 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:44.656459 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:44.657635 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0928 15:53:44.657893 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1294,"thread_start_us":1166,"threads_started":1}
Sep 28 15:53:44 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790610824, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:44.661697 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:44.630905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:42647 (local address 127.22.157.194:38923)
0928 15:53:44.634731 (+ 3826us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:44.634791 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:44.637513 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:44.638727 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:44.638739 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:44.639493 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:44.640453 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:44.640482 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:44.642187 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:44.642193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:44.644190 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:44.644207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:44.645747 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:44.645755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:44.646101 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:44.646958 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:44.646993 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:44.649969 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:44.654241 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:44.654252 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:44.654272 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:44.654936 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:44.655423 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:44.655428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:44.655431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:44.655768 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:44.656579 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:44.656587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:44.657145 (+ 558us) client_negotiation.cc:769] Sending connection context
0928 15:53:44.659348 (+ 2203us) client_negotiation.cc:240] Negotiation successful
0928 15:53:44.659984 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3464,"thread_start_us":3206,"threads_started":1}
I20260928 15:53:44.663854 4100 heartbeater.cc:347] Connected to a master server at 127.22.157.254:42647
I20260928 15:53:44.664223 4100 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:44.665233 4100 heartbeater.cc:511] Master 127.22.157.254:42647 requested a full tablet report, sending...
I20260928 15:53:44.667517 3756 ts_manager.cc:194] Registered new tserver with Master: 4db47406d5f946bdb454ee797d48dabb (127.22.157.194:34463)
I20260928 15:53:44.669951 3756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:38923
W20260928 15:53:45.040800 4103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:45.041290 4103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:45.041496 4103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 15:53:45.049557 3955 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:45.050001 3955 heartbeater.cc:511] Master 127.22.157.254:42647 requested a full tablet report, sending...
I20260928 15:53:45.051354 3756 ts_manager.cc:194] Registered new tserver with Master: 3128ea3052a74d40b9d9ff741c19bb29 (127.22.157.193:35673)
I20260928 15:53:45.052524 3756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:57215
W20260928 15:53:45.053308 4103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:45.053469 4103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:45.053681 4103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:53:45.069391 4103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.157.212:58025/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:35255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=40011
--tserver_master_addrs=127.22.157.254:42647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:45.071324 4103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:45.073201 4103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:45.090158 4110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:45.092072 4111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:45.099255 4113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:45.100615 4103 server_base.cc:1034] running on GCE node
I20260928 15:53:45.101604 4103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:45.103046 4103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:45.104370 4103 hybrid_clock.cc:648] HybridClock initialized: now 1790610825104294 us; error 61 us; skew 500 ppm
I20260928 15:53:45.104873 4103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:45 dist-test-slave-1vw0 krb5kdc[2735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610825, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:45.109344 4103 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:53:45.115010 4103 webserver.cc:460] Webserver started at http://127.22.157.195:40011/ using document root <none> and password file <none>
I20260928 15:53:45.116125 4103 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:45.116362 4103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 15:53:45 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610825, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:45.160069 4103 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.004s sys 0.000s
Sep 28 15:53:45 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610824, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:45.176869 4121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:45.179176 4103 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.004s sys 0.010s
I20260928 15:53:45.179483 4103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "a11921c6a91e4bbebf605354f19c53de"
format_stamp: "Formatted at 2026-09-28 15:53:42 on dist-test-slave-1vw0"
server_key: "c90565487cacbb235005dba6a73edc4b"
server_key_iv: "a0eb9c68c032301c3f20bcc579d05984"
server_key_version: "kuduclusterkey@0"
I20260928 15:53:45.181195 4103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:45.258524 4103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:45.264358 4103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:45.264796 4103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:45.277305 4103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:53:45.285730 4103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:53:45.285959 4103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:53:45.286188 4103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:53:45.286454 4103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 15:53:45 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610824, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for HTTP/127.22.157.212@KRBTEST.COM
I20260928 15:53:45.675663 4100 heartbeater.cc:503] Master 127.22.157.254:42647 was elected leader, sending a full tablet report...
I20260928 15:53:45.965206 4103 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:35255
I20260928 15:53:45.965266 4242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:35255 every 8 connection(s)
I20260928 15:53:45.967172 4103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:53:45.974731 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 4103
Sep 28 15:53:46 dist-test-slave-1vw0 krb5kdc[2735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610825, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:46.022629 4246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:45.971767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:53645 (local address 127.22.157.254:42647)
0928 15:53:45.994838 (+ 23071us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:45.994846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.000658 (+ 5812us) server_negotiation.cc:407] Connection header received
0928 15:53:46.003373 (+ 2715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.003379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.003463 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.003581 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:46.005968 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.007128 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.009577 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.010206 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.010305 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.014860 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:46.014866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:46.014871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:46.014935 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:46.018691 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.019710 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.019716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.019720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.019864 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.020430 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.020435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.020437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.020846 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:46.020974 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.021999 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.022223 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22905,"thread_start_us":31967,"threads_started":1}
I20260928 15:53:46.023126 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:45.976220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.22.157.254:42647)
0928 15:53:46.002891 (+ 26671us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.002897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.003731 (+ 834us) server_negotiation.cc:407] Connection header received
0928 15:53:46.003789 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.003794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.003873 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.003962 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:46.006355 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.007585 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.009662 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.010421 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.010526 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.014429 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:46.014435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:46.014440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:46.014569 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:46.019070 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.020062 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.020068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.020072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.020219 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.020657 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.020663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.020666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.021037 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:46.021322 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.022543 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.022761 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26529,"thread_start_us":26683,"threads_started":1}
I20260928 15:53:46.025218 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:45.971086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:42647 (local address 127.22.157.195:53645)
0928 15:53:45.995475 (+ 24389us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:45.995556 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:46.002857 (+ 7301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:46.004165 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:46.004178 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:46.004833 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:46.005758 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.005786 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.007338 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.007343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.009393 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.009421 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.010693 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.010700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.010984 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.011812 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:46.011842 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:46.014572 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:46.018888 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:46.018899 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:46.018920 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:46.019567 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:46.020008 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:46.020012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:46.020016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:46.020299 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:46.021079 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:46.021088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:46.021570 (+ 482us) client_negotiation.cc:769] Sending connection context
0928 15:53:46.023175 (+ 1605us) client_negotiation.cc:240] Negotiation successful
0928 15:53:46.023755 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24019,"thread_start_us":28440,"threads_started":1}
I20260928 15:53:46.027081 4243 heartbeater.cc:347] Connected to a master server at 127.22.157.254:42647
I20260928 15:53:46.027441 4243 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:46.028388 4243 heartbeater.cc:511] Master 127.22.157.254:42647 requested a full tablet report, sending...
I20260928 15:53:46.030099 3756 ts_manager.cc:194] Registered new tserver with Master: a11921c6a91e4bbebf605354f19c53de (127.22.157.195:35255)
I20260928 15:53:46.031184 3756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:53645
I20260928 15:53:46.038870 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:53:46.056388 3955 heartbeater.cc:503] Master 127.22.157.254:42647 was elected leader, sending a full tablet report...
I20260928 15:53:46.061041 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.042339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.22.157.254:42647)
0928 15:53:46.042518 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.042524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.044957 (+ 2433us) server_negotiation.cc:407] Connection header received
0928 15:53:46.045191 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.045196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.045313 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.045422 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:46.047514 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.048600 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.050465 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.051056 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.051181 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.054435 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:46.054442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:46.054447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:46.054498 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:46.057985 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.058826 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.058832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.058835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.058984 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.059417 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.059421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.059424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.059786 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:46.060021 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.060471 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.060703 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:53:46.066990 3756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:53:46.077759 3756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:53:46.132778 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.104647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.22.157.194:34463)
0928 15:53:46.121933 (+ 17286us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.121943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.122888 (+ 945us) server_negotiation.cc:407] Connection header received
0928 15:53:46.123034 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.123044 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.123484 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.123749 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:46.125202 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.126809 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.129586 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.131037 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.131225 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.131398 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.132204 (+ 806us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.132375 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17087,"thread_start_us":17294,"threads_started":1}
I20260928 15:53:46.134286 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.104267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:34463 (local address 127.0.0.1:60440)
0928 15:53:46.122610 (+ 18343us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:46.122629 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:46.122754 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:46.124287 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:46.124299 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:46.124359 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:46.124884 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.124899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.126999 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.127008 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.129382 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.129398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.131586 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.131592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.131877 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.131933 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 15:53:46.132051 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:53:46.133723 (+ 1672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18222,"mutex_wait_us":1,"thread_start_us":15679,"threads_started":1}
I20260928 15:53:46.135752 4260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.099647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:35255 (local address 127.0.0.1:45452)
0928 15:53:46.120426 (+ 20779us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:46.120468 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:46.120660 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:46.126704 (+ 6044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:46.126708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:46.126729 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:46.127112 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.127125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.130297 (+ 3172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.130302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.132616 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.132632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.134981 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.134987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.135157 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.135190 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:53:46.135288 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:53:46.135414 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20478,"thread_start_us":26489,"threads_started":1}
I20260928 15:53:46.136526 4259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.099924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.22.157.195:35255)
0928 15:53:46.124961 (+ 25037us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.124970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.125002 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:53:46.125109 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.125116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.125372 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.125595 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:46.127321 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.128931 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.132834 (+ 3903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.134402 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.134609 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.134791 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.135388 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.135651 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24840,"thread_start_us":25902,"threads_started":1}
I20260928 15:53:46.139278 4034 tablet_service.cc:1467] Processing CreateTablet for tablet f9c81ef25e93479a81c55161b22ff063 (DEFAULT_TABLE table=test-table [id=6a163a6933a0447cbd0a7c41d407fd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:53:46.140601 4169 tablet_service.cc:1467] Processing CreateTablet for tablet f9c81ef25e93479a81c55161b22ff063 (DEFAULT_TABLE table=test-table [id=6a163a6933a0447cbd0a7c41d407fd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:53:46.141387 4034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c81ef25e93479a81c55161b22ff063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:46.142138 4169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c81ef25e93479a81c55161b22ff063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:46.154541 4258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.121092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35673 (local address 127.0.0.1:56302)
0928 15:53:46.126438 (+ 5346us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:46.126470 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:46.126573 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:46.146596 (+ 20023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:46.146603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:46.146642 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:46.147013 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.147028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.149444 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.149451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.151621 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.151643 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.153670 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.153678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.153862 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.153901 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 15:53:46.153974 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 15:53:46.154167 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5209,"thread_start_us":1932,"threads_started":1}
I20260928 15:53:46.155807 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.121493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.22.157.193:35673)
0928 15:53:46.145664 (+ 24171us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.145674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.145709 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:53:46.145822 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.145830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.146100 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.147262 (+ 1162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:46.147512 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.149269 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.151868 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.153404 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.155029 (+ 1625us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.155250 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.155323 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.155490 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23954,"thread_start_us":24777,"threads_started":1}
I20260928 15:53:46.159858 3889 tablet_service.cc:1467] Processing CreateTablet for tablet f9c81ef25e93479a81c55161b22ff063 (DEFAULT_TABLE table=test-table [id=6a163a6933a0447cbd0a7c41d407fd3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:53:46.161417 3889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c81ef25e93479a81c55161b22ff063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:46.164413 4264 tablet_bootstrap.cc:492] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: Bootstrap starting.
I20260928 15:53:46.170974 4264 tablet_bootstrap.cc:654] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:46.172495 4264 log.cc:824] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:46.181551 4265 tablet_bootstrap.cc:492] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: Bootstrap starting.
I20260928 15:53:46.182741 4264 tablet_bootstrap.cc:492] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: No bootstrap required, opened a new log
I20260928 15:53:46.183197 4264 ts_tablet_manager.cc:1397] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.011s
I20260928 15:53:46.186719 4265 tablet_bootstrap.cc:654] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:46.188532 4265 log.cc:824] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:46.193915 4264 raft_consensus.cc:348] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.194602 4264 raft_consensus.cc:374] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:46.194761 4264 raft_consensus.cc:729] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db47406d5f946bdb454ee797d48dabb, State: Initialized, Role: FOLLOWER
I20260928 15:53:46.195606 4264 consensus_queue.cc:260] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.198200 4267 tablet_bootstrap.cc:492] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: Bootstrap starting.
I20260928 15:53:46.202435 4267 tablet_bootstrap.cc:654] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:46.203552 4267 log.cc:824] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:46.203840 4264 ts_tablet_manager.cc:1428] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: Time spent starting tablet: real 0.020s user 0.009s sys 0.010s
I20260928 15:53:46.207705 4265 tablet_bootstrap.cc:492] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: No bootstrap required, opened a new log
I20260928 15:53:46.208079 4265 ts_tablet_manager.cc:1397] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20260928 15:53:46.214310 4267 tablet_bootstrap.cc:492] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: No bootstrap required, opened a new log
I20260928 15:53:46.214617 4267 ts_tablet_manager.cc:1397] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260928 15:53:46.224009 4265 raft_consensus.cc:348] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.225140 4267 raft_consensus.cc:348] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.225662 4267 raft_consensus.cc:374] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:46.225764 4267 raft_consensus.cc:729] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3128ea3052a74d40b9d9ff741c19bb29, State: Initialized, Role: FOLLOWER
I20260928 15:53:46.226476 4267 consensus_queue.cc:260] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.235082 4267 ts_tablet_manager.cc:1428] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260928 15:53:46.235414 4265 raft_consensus.cc:374] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:46.235600 4265 raft_consensus.cc:729] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11921c6a91e4bbebf605354f19c53de, State: Initialized, Role: FOLLOWER
I20260928 15:53:46.236418 4265 consensus_queue.cc:260] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.240773 4243 heartbeater.cc:503] Master 127.22.157.254:42647 was elected leader, sending a full tablet report...
I20260928 15:53:46.241676 4265 ts_tablet_manager.cc:1428] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: Time spent starting tablet: real 0.033s user 0.017s sys 0.000s
W20260928 15:53:46.252569 3956 tablet.cc:2369] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:53:46.387434 4104 tablet.cc:2369] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:53:46.458954 4272 raft_consensus.cc:482] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:53:46.459250 4272 raft_consensus.cc:504] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.460943 4272 leader_election.cc:288] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4db47406d5f946bdb454ee797d48dabb (127.22.157.194:34463), 3128ea3052a74d40b9d9ff741c19bb29 (127.22.157.193:35673)
I20260928 15:53:46.468218 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.461369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:55995 (local address 127.22.157.194:34463)
0928 15:53:46.461615 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.461620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.461686 (+ 66us) server_negotiation.cc:407] Connection header received
0928 15:53:46.461939 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.461946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.462027 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.462155 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:46.462571 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.463553 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.466091 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.467362 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.467503 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.467620 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.467915 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.468034 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 15:53:46.468304 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.461386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:34463 (local address 127.22.157.195:55995)
0928 15:53:46.461580 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:46.461606 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:46.461755 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:46.462178 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:46.462183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:46.462213 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:46.462429 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.462442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.463789 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.463796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.465897 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.465911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.467511 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.467517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.467724 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.467760 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:53:46.467863 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:53:46.468009 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":84}
I20260928 15:53:46.469755 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.462210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:47343 (local address 127.22.157.193:35673)
0928 15:53:46.462459 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.462465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.462836 (+ 371us) server_negotiation.cc:407] Connection header received
0928 15:53:46.462963 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.462967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.463042 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.463139 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:46.463716 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.464834 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.467456 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.468804 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.468935 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.468984 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.469373 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.469467 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 15:53:46.469887 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.461845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35673 (local address 127.22.157.195:47343)
0928 15:53:46.462725 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:46.462744 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:46.462843 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:46.463192 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:46.463196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:46.463219 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:46.463529 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.463537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.465023 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.465028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.467235 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.467253 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.469017 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.469023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.469192 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.469227 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:53:46.469340 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 15:53:46.469505 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":3,"thread_start_us":630,"threads_started":1}
I20260928 15:53:46.469846 4271 raft_consensus.cc:482] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:53:46.469849 4054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c81ef25e93479a81c55161b22ff063" candidate_uuid: "a11921c6a91e4bbebf605354f19c53de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db47406d5f946bdb454ee797d48dabb" is_pre_election: true
I20260928 15:53:46.470120 4271 raft_consensus.cc:504] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.470353 4054 raft_consensus.cc:2454] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a11921c6a91e4bbebf605354f19c53de in term 0.
I20260928 15:53:46.471763 4271 leader_election.cc:288] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a11921c6a91e4bbebf605354f19c53de (127.22.157.195:35255), 4db47406d5f946bdb454ee797d48dabb (127.22.157.194:34463)
I20260928 15:53:46.472256 4122 leader_election.cc:302] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db47406d5f946bdb454ee797d48dabb, a11921c6a91e4bbebf605354f19c53de; no voters:
I20260928 15:53:46.473531 3909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c81ef25e93479a81c55161b22ff063" candidate_uuid: "a11921c6a91e4bbebf605354f19c53de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3128ea3052a74d40b9d9ff741c19bb29" is_pre_election: true
I20260928 15:53:46.474085 3909 raft_consensus.cc:2454] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a11921c6a91e4bbebf605354f19c53de in term 0.
I20260928 15:53:46.474370 4272 raft_consensus.cc:2781] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:53:46.474547 4272 raft_consensus.cc:482] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:53:46.474653 4272 raft_consensus.cc:3037] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 0 FOLLOWER]: Advancing to term 1
W20260928 15:53:46.474960 4244 tablet.cc:2369] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:53:46.477380 4272 raft_consensus.cc:504] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.478154 4272 leader_election.cc:288] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [CANDIDATE]: Term 1 election: Requested vote from peers 4db47406d5f946bdb454ee797d48dabb (127.22.157.194:34463), 3128ea3052a74d40b9d9ff741c19bb29 (127.22.157.193:35673)
I20260928 15:53:46.479084 4054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c81ef25e93479a81c55161b22ff063" candidate_uuid: "a11921c6a91e4bbebf605354f19c53de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db47406d5f946bdb454ee797d48dabb"
I20260928 15:53:46.479311 4054 raft_consensus.cc:3037] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:46.479629 4259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.472024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:60039 (local address 127.22.157.195:35255)
0928 15:53:46.472229 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.472235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.473188 (+ 953us) server_negotiation.cc:407] Connection header received
0928 15:53:46.473456 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.473467 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.473522 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.473615 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:46.474226 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.475368 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.477611 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.478585 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.478732 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.478801 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.479257 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.479379 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:53:46.479754 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.472101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:35255 (local address 127.22.157.193:60039)
0928 15:53:46.473080 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:46.473102 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:46.473303 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:46.473629 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:46.473633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:46.473660 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:46.474064 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.474079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.475509 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.475514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.477463 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.477474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.478930 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.478935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.479098 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.479128 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 15:53:46.479235 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 15:53:46.479369 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":22,"thread_start_us":530,"threads_started":1}
I20260928 15:53:46.480533 4190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c81ef25e93479a81c55161b22ff063" candidate_uuid: "3128ea3052a74d40b9d9ff741c19bb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a11921c6a91e4bbebf605354f19c53de" is_pre_election: true
I20260928 15:53:46.480973 4190 raft_consensus.cc:2379] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3128ea3052a74d40b9d9ff741c19bb29 in current term 1: Already voted for candidate a11921c6a91e4bbebf605354f19c53de in this term.
I20260928 15:53:46.482084 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.472519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:37405 (local address 127.22.157.194:34463)
0928 15:53:46.472697 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.472704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.473431 (+ 727us) server_negotiation.cc:407] Connection header received
0928 15:53:46.473701 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.473707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.473784 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.473847 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:46.475732 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.476879 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.479355 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.480778 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.480937 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.481017 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.481689 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.481837 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:53:46.482182 4274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.472122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:34463 (local address 127.22.157.193:37405)
0928 15:53:46.473338 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:46.473355 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:46.473528 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:46.474046 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:46.474050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:46.474080 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:46.474591 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.474617 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.477027 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.477033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.479175 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:46.479189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.481206 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.481213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:46.481469 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.481501 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:53:46.481634 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 15:53:46.481777 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":28,"thread_start_us":925,"threads_started":1}
I20260928 15:53:46.482888 4054 raft_consensus.cc:2454] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a11921c6a91e4bbebf605354f19c53de in term 1.
I20260928 15:53:46.482976 4053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c81ef25e93479a81c55161b22ff063" candidate_uuid: "3128ea3052a74d40b9d9ff741c19bb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db47406d5f946bdb454ee797d48dabb" is_pre_election: true
I20260928 15:53:46.483348 4122 leader_election.cc:302] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db47406d5f946bdb454ee797d48dabb, a11921c6a91e4bbebf605354f19c53de; no voters:
I20260928 15:53:46.483402 4053 raft_consensus.cc:2379] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3128ea3052a74d40b9d9ff741c19bb29 in current term 1: Already voted for candidate a11921c6a91e4bbebf605354f19c53de in this term.
I20260928 15:53:46.483525 3909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c81ef25e93479a81c55161b22ff063" candidate_uuid: "a11921c6a91e4bbebf605354f19c53de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3128ea3052a74d40b9d9ff741c19bb29"
I20260928 15:53:46.483675 4272 raft_consensus.cc:2781] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:53:46.483808 3909 raft_consensus.cc:3037] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:46.484192 4272 raft_consensus.cc:686] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 1 LEADER]: Becoming Leader. State: Replica: a11921c6a91e4bbebf605354f19c53de, State: Running, Role: LEADER
I20260928 15:53:46.484663 4272 consensus_queue.cc:237] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } }
I20260928 15:53:46.486346 3909 raft_consensus.cc:2454] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a11921c6a91e4bbebf605354f19c53de in term 1.
I20260928 15:53:46.487280 3841 leader_election.cc:302] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3128ea3052a74d40b9d9ff741c19bb29; no voters: 4db47406d5f946bdb454ee797d48dabb, a11921c6a91e4bbebf605354f19c53de
I20260928 15:53:46.487695 4271 raft_consensus.cc:2726] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 15:53:46.489398 3756 catalog_manager.cc:5495] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de reported cstate change: term changed from 0 to 1, leader changed from <none> to a11921c6a91e4bbebf605354f19c53de (127.22.157.195). New cstate: current_term: 1 leader_uuid: "a11921c6a91e4bbebf605354f19c53de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a11921c6a91e4bbebf605354f19c53de" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 35255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 } health_report { overall_health: UNKNOWN } } }
I20260928 15:53:46.567675 4259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.561566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.22.157.195:35255)
0928 15:53:46.561749 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.561754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.562198 (+ 444us) server_negotiation.cc:407] Connection header received
0928 15:53:46.562395 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.562399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.562475 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.562602 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:53:46.562968 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.563745 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.565354 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.565817 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.565979 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.566524 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:53:46.567046 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:53:46.567214 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.567281 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.567365 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:53:46.576262 4053 raft_consensus.cc:1261] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 1 FOLLOWER]: Refusing update from remote peer a11921c6a91e4bbebf605354f19c53de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:53:46.576313 3909 raft_consensus.cc:1261] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 1 FOLLOWER]: Refusing update from remote peer a11921c6a91e4bbebf605354f19c53de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:53:46.577196 4276 consensus_queue.cc:1035] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [LEADER]: Connected to new peer: Peer: permanent_uuid: "3128ea3052a74d40b9d9ff741c19bb29" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:53:46.577591 4272 consensus_queue.cc:1035] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db47406d5f946bdb454ee797d48dabb" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:53:46.604132 3755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:53:46.604364 3755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:53:46.607885 3755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a732e0db6809427b9939c5adc7a27199: Sending DeleteTablet for 3 replicas of tablet f9c81ef25e93479a81c55161b22ff063
I20260928 15:53:46.609057 4169 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c81ef25e93479a81c55161b22ff063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:53:46 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:45452
I20260928 15:53:46.609246 3889 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c81ef25e93479a81c55161b22ff063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:53:46 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:56302
I20260928 15:53:46.609246 4034 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c81ef25e93479a81c55161b22ff063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:53:46 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:60440
I20260928 15:53:46.612618 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 3793
I20260928 15:53:46.625262 4290 tablet_replica.cc:330] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: stopping tablet replica
I20260928 15:53:46.625934 4290 raft_consensus.cc:2229] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:53:46.626773 4290 raft_consensus.cc:2258] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:53:46.629287 4290 ts_tablet_manager.cc:1905] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:53:46.637809 4291 tablet_replica.cc:330] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: stopping tablet replica
I20260928 15:53:46.638412 4290 ts_tablet_manager.cc:1918] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:53:46.639107 4290 log.cc:1196] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/f9c81ef25e93479a81c55161b22ff063
I20260928 15:53:46.639592 4291 raft_consensus.cc:2229] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:53:46.639901 4290 ts_tablet_manager.cc:1939] T f9c81ef25e93479a81c55161b22ff063 P a11921c6a91e4bbebf605354f19c53de: Deleting consensus metadata
I20260928 15:53:46.640249 4291 raft_consensus.cc:2258] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:53:46.640821 4292 tablet_replica.cc:330] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: stopping tablet replica
I20260928 15:53:46.641314 4292 raft_consensus.cc:2229] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:53:46.641834 4292 raft_consensus.cc:2258] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:53:46.641984 3742 catalog_manager.cc:4841] TS a11921c6a91e4bbebf605354f19c53de (127.22.157.195:35255): tablet f9c81ef25e93479a81c55161b22ff063 (table test-table [id=6a163a6933a0447cbd0a7c41d407fd3f]) successfully deleted
I20260928 15:53:46.644433 4292 ts_tablet_manager.cc:1905] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:53:46.644433 4291 ts_tablet_manager.cc:1905] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:53:46.652320 4292 ts_tablet_manager.cc:1918] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:53:46.652429 4291 ts_tablet_manager.cc:1918] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:53:46.652629 4292 log.cc:1196] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/f9c81ef25e93479a81c55161b22ff063
I20260928 15:53:46.653159 4291 log.cc:1196] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/f9c81ef25e93479a81c55161b22ff063
I20260928 15:53:46.653632 4292 ts_tablet_manager.cc:1939] T f9c81ef25e93479a81c55161b22ff063 P 3128ea3052a74d40b9d9ff741c19bb29: Deleting consensus metadata
I20260928 15:53:46.655699 3740 catalog_manager.cc:4841] TS 3128ea3052a74d40b9d9ff741c19bb29 (127.22.157.193:35673): tablet f9c81ef25e93479a81c55161b22ff063 (table test-table [id=6a163a6933a0447cbd0a7c41d407fd3f]) successfully deleted
I20260928 15:53:46.656032 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.613917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.22.157.193:35673)
0928 15:53:46.614159 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.614169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.633164 (+ 18995us) server_negotiation.cc:407] Connection header received
0928 15:53:46.633512 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.633521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.633651 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.633748 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:46.638778 (+ 5030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.640012 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.641985 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.642704 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.642815 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.646926 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:46.646941 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:46.646960 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:46.647017 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:46.651440 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.652553 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.652564 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.652568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.652803 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.653269 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.653276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.653282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.653806 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:46.654137 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.654853 (+ 716us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.655169 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:53:46.657728 3949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:46.658505 4291 ts_tablet_manager.cc:1939] T f9c81ef25e93479a81c55161b22ff063 P 4db47406d5f946bdb454ee797d48dabb: Deleting consensus metadata
I20260928 15:53:46.661067 3740 catalog_manager.cc:4841] TS 4db47406d5f946bdb454ee797d48dabb (127.22.157.194:34463): tablet f9c81ef25e93479a81c55161b22ff063 (table test-table [id=6a163a6933a0447cbd0a7c41d407fd3f]) successfully deleted
I20260928 15:53:46.756268 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 3793
I20260928 15:53:46.775202 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 3959
I20260928 15:53:46.798197 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.776132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.22.157.194:34463)
0928 15:53:46.776316 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.776323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.776366 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:53:46.776630 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.776653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.776777 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.776908 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:46.778266 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.779002 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.780792 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.781430 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.781575 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.791516 (+ 9941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:46.791534 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:46.791551 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:46.791605 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:46.794872 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.795775 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.795789 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.795793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.795975 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.796416 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.796424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.796436 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.796913 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:46.797282 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.797593 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.797846 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:53:46.798872 4094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:46.891520 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 3959
I20260928 15:53:46.909852 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 4103
I20260928 15:53:46.927481 4259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:46.910742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.22.157.195:35255)
0928 15:53:46.910987 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:46.910992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:46.911009 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:53:46.911268 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:46.911273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:46.911353 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:46.911472 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:46.913416 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.914249 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.916277 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:46.916908 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:46.917031 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:46.920479 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:46.920503 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:46.920519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:46.920578 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:46.924033 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.925044 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.925052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.925054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.925216 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:46.925714 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:46.925719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:46.925721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:46.926246 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:46.926554 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:46.926962 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 15:53:46.927178 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 15:53:46.928303 4237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:47.077684 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 4103
I20260928 15:53:47.097867 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 3722
I20260928 15:53:47.099241 3786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:47.215854 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 3722
I20260928 15:53:47.241991 23159 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 15:53:47.588404 23159 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 15:53:47.588778 23159 mini_ranger.cc:66] Stopping Ranger...
I20260928 15:53:47.935066 23159 mini_ranger.cc:68] Stopped Ranger
2026-09-28 15:53:47.935 UTC [2765] LOG: received smart shutdown request
2026-09-28 15:53:47.939 UTC [2765] LOG: background worker "logical replication launcher" (PID 2774) exited with exit code 1
2026-09-28 15:53:47.939 UTC [2769] LOG: shutting down
2026-09-28 15:53:48.147 UTC [2765] LOG: database system is shut down
2026-09-28T15:53:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54780 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:53:48 dist-test-slave-1vw0 krb5kdc[4306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:53:48 dist-test-slave-1vw0 krb5kdc[4306](info): set up 2 sockets
Sep 28 15:53:48 dist-test-slave-1vw0 krb5kdc[4306](info): commencing operation
krb5kdc: starting...
W20260928 15:53:49.425730 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:53:49 dist-test-slave-1vw0 krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:53:49.701520 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:40711
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:40711,127.22.157.253:33815,127.22.157.252:45887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:53:50.058141 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:50.058514 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:50.058687 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:50.066272 4319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:53:50.066447 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:50.066572 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:50.066676 4319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:53:50.066767 4319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:53:50.077279 4319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:40711,127.22.157.253:33815,127.22.157.252:45887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:40711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:50.078763 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:50.080153 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:50.093129 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:50.093202 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:50.093166 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:50.094697 4319 server_base.cc:1034] running on GCE node
I20260928 15:53:50.095609 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:50.097263 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:50.098599 4319 hybrid_clock.cc:648] HybridClock initialized: now 1790610830098534 us; error 76 us; skew 500 ppm
I20260928 15:53:50.099020 4319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:50 dist-test-slave-1vw0 krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:50.102579 4319 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:53:50.104277 4319 webserver.cc:460] Webserver started at http://127.22.157.254:42055/ using document root <none> and password file <none>
I20260928 15:53:50.105111 4319 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:50.105260 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:50.105592 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:53:50.108798 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "41f397d35a984790bb467c27f19ab94e"
format_stamp: "Formatted at 2026-09-28 15:53:50 on dist-test-slave-1vw0"
server_key: "4cd4967bea15799e506fe0fd9f5105ba"
server_key_iv: "7a1719ad569bbdcdcb9af02f4cbb995f"
server_key_version: "encryptionkey@0"
I20260928 15:53:50.109680 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "41f397d35a984790bb467c27f19ab94e"
format_stamp: "Formatted at 2026-09-28 15:53:50 on dist-test-slave-1vw0"
server_key: "4cd4967bea15799e506fe0fd9f5105ba"
server_key_iv: "7a1719ad569bbdcdcb9af02f4cbb995f"
server_key_version: "encryptionkey@0"
I20260928 15:53:50.115451 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:53:50.119730 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:50.121573 4319 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:53:50.121807 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "41f397d35a984790bb467c27f19ab94e"
format_stamp: "Formatted at 2026-09-28 15:53:50 on dist-test-slave-1vw0"
server_key: "4cd4967bea15799e506fe0fd9f5105ba"
server_key_iv: "7a1719ad569bbdcdcb9af02f4cbb995f"
server_key_version: "encryptionkey@0"
I20260928 15:53:50.122046 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:50.156584 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:50.161710 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:50.162070 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:50.182823 4319 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:40711
I20260928 15:53:50.182869 4385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:40711 every 8 connection(s)
I20260928 15:53:50.184343 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:53:50.188429 4386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:50.189152 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 4319
I20260928 15:53:50.189384 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:53:50.190088 23159 external_mini_cluster.cc:1428] Setting key 66febc51c03f53b47a45cad7b57b2f90
I20260928 15:53:50.196038 4386 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.157.253@KRBTEST.COM; defaulting to no policy
Sep 28 15:53:50 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.253@KRBTEST.COM" created.
I20260928 15:53:50.224429 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.199731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:40711 (local address 127.0.0.1:47810)
0928 15:53:50.200684 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.200766 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:50.203762 (+ 2996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:50.204921 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:50.204933 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:50.205884 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:50.206451 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:50.206483 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.208210 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.208219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:50.210424 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:50.210450 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.211671 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.211682 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:50.211961 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:50.212840 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:50.212874 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:50.215827 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:50.219972 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:50.219984 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:50.219996 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:50.220449 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:50.221013 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:50.221017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:50.221019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:50.221193 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:50.222337 (+ 1144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:50.222343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:50.222551 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 15:53:50.222767 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 15:53:50.223224 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":327,"threads_started":1}
I20260928 15:53:50.224538 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.200952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.22.157.254:40711)
0928 15:53:50.201577 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:50.201583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:50.201632 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:53:50.204317 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:50.204323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:50.204590 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:50.204869 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:50.206717 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.208014 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.210694 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.211341 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.211508 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:50.216100 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:50.216109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:50.216123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:50.216154 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:50.219773 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:50.220619 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:50.220631 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:50.220637 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:50.220883 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:50.221318 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:50.221325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:50.221331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:50.222180 (+ 849us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:50.222391 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:50.222974 (+ 583us) server_negotiation.cc:299] Negotiation successful
0928 15:53:50.223458 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":453,"thread_start_us":309,"threads_started":1}
I20260928 15:53:50.228207 4386 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:53:50.229666 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.229118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46634)
0928 15:53:50.229295 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.229543 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 15:53:50.229976 4338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
I20260928 15:53:50.231639 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.231264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46648)
0928 15:53:50.231416 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.231502 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:53:50.231866 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:33815: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab.
Entry for principal kudu/127.22.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab.
I20260928 15:53:50.252086 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.253:33815
--webserver_interface=127.22.157.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab
--principal=kudu/127.22.157.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:40711,127.22.157.253:33815,127.22.157.252:45887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:53:50.281198 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } attempt: 1
I20260928 15:53:50.282578 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.281959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46650)
0928 15:53:50.282229 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.282377 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 15:53:50.283725 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.283308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46666)
0928 15:53:50.283475 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.283565 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:53:50.283947 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:33815: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
I20260928 15:53:50.352239 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } attempt: 2
I20260928 15:53:50.353734 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.353164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46674)
0928 15:53:50.353469 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.353576 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 15:53:50.355047 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.354618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46676)
0928 15:53:50.354793 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.354883 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 15:53:50.355311 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:33815: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
I20260928 15:53:50.446604 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } attempt: 3
I20260928 15:53:50.447929 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.447417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46678)
0928 15:53:50.447676 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.447777 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 15:53:50.449437 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.448991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46690)
0928 15:53:50.449175 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.449280 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 15:53:50.449744 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:33815: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
I20260928 15:53:50.593048 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } attempt: 4
I20260928 15:53:50.594758 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.593870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46704)
0928 15:53:50.594368 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.594535 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
I20260928 15:53:50.595927 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.595504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46708)
0928 15:53:50.595676 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.595766 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260928 15:53:50.596127 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:33815: Network error: Client connection negotiation failed: client connection to 127.22.157.253:33815: connect: Connection refused (error 111)
W20260928 15:53:50.602547 4393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:50.602917 4393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:50.603086 4393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:50.610606 4393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:53:50.610781 4393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:50.610903 4393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:50.611003 4393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:53:50.611091 4393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:53:50.621797 4393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:40711,127.22.157.253:33815,127.22.157.252:45887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab
--principal=kudu/127.22.157.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.253:33815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.157.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:50.623312 4393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:50.624886 4393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:50.638432 4399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:50.638479 4398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:50.638919 4401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:50.639267 4393 server_base.cc:1034] running on GCE node
I20260928 15:53:50.640164 4393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:50.641803 4393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:50.643100 4393 hybrid_clock.cc:648] HybridClock initialized: now 1790610830643035 us; error 50 us; skew 500 ppm
I20260928 15:53:50.643550 4393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:50 dist-test-slave-1vw0 krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:50.647330 4393 init.cc:377] Logged in from keytab as kudu/127.22.157.253@KRBTEST.COM (short username kudu)
I20260928 15:53:50.649186 4393 webserver.cc:460] Webserver started at http://127.22.157.253:43557/ using document root <none> and password file <none>
I20260928 15:53:50.649922 4393 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:50.650048 4393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:50.650338 4393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:53:50.653976 4393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/data/instance:
uuid: "c264fe1c6af646648913367a481adcf4"
format_stamp: "Formatted at 2026-09-28 15:53:50 on dist-test-slave-1vw0"
server_key: "2bdadea72df297a12dbf585e385faedb"
server_key_iv: "9ceacdd9f06b9a82ec2ffa395e190ebe"
server_key_version: "encryptionkey@0"
I20260928 15:53:50.654942 4393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/wal/instance:
uuid: "c264fe1c6af646648913367a481adcf4"
format_stamp: "Formatted at 2026-09-28 15:53:50 on dist-test-slave-1vw0"
server_key: "2bdadea72df297a12dbf585e385faedb"
server_key_iv: "9ceacdd9f06b9a82ec2ffa395e190ebe"
server_key_version: "encryptionkey@0"
I20260928 15:53:50.661159 4393 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 15:53:50.664844 4408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:50.666308 4393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:53:50.666501 4393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/wal
uuid: "c264fe1c6af646648913367a481adcf4"
format_stamp: "Formatted at 2026-09-28 15:53:50 on dist-test-slave-1vw0"
server_key: "2bdadea72df297a12dbf585e385faedb"
server_key_iv: "9ceacdd9f06b9a82ec2ffa395e190ebe"
server_key_version: "encryptionkey@0"
I20260928 15:53:50.666713 4393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:50.682379 4393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:50.686679 4393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:50.686987 4393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:50.707409 4393 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.253:33815
I20260928 15:53:50.707501 4461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.253:33815 every 8 connection(s)
I20260928 15:53:50.708891 4393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
I20260928 15:53:50.712838 4462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:50.718627 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 4393
I20260928 15:53:50.718858 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-1/wal/instance
I20260928 15:53:50.719434 23159 external_mini_cluster.cc:1428] Setting key 01f0f48d07d8bd8b07957274127584f1
I20260928 15:53:50.721120 4462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:53:50 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.252@KRBTEST.COM; defaulting to no policy
I20260928 15:53:50.747998 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.725254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.22.157.254:40711)
0928 15:53:50.725509 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:50.725518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:50.726926 (+ 1408us) server_negotiation.cc:407] Connection header received
0928 15:53:50.730196 (+ 3270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:50.730205 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:50.730316 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:50.730449 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:50.732131 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.733368 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.735093 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.735894 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.736057 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:50.740033 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:50.740042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:50.740050 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:50.740102 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:50.744016 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:50.745077 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:50.745086 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:50.745091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:50.745248 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:50.746011 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:50.746016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:50.746018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:50.746332 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:50.746510 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:50.747509 (+ 999us) server_negotiation.cc:299] Negotiation successful
0928 15:53:50.747751 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.252@KRBTEST.COM" created.
I20260928 15:53:50.749379 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.725831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:40711 (local address 127.0.0.1:47814)
0928 15:53:50.726726 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.726784 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:50.729819 (+ 3035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:50.730685 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:50.730694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:50.731282 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:50.731905 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:50.731924 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.733563 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.733567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:50.734919 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:50.734933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.736155 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.736160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:50.736380 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:50.737208 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:50.737232 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:50.739756 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:50.744237 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:50.744246 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:50.744263 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:50.744912 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:50.745385 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:50.745389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:50.745391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:50.745879 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:50.746538 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:50.746548 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:50.747028 (+ 480us) client_negotiation.cc:769] Sending connection context
0928 15:53:50.747323 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 15:53:50.747862 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":355,"threads_started":1}
I20260928 15:53:50.751412 4462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:53:50 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
I20260928 15:53:50.767684 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.752265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46710)
0928 15:53:50.752501 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.752524 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:50.752654 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:50.753836 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:50.753844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:50.754467 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:50.754701 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:50.754717 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.756154 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.756162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:50.757728 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:50.757748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.758713 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.758719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:50.758937 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:50.759584 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:50.759601 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:50.761773 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:50.765144 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:50.765149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:50.765153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:50.765517 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:50.765904 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:50.765912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:50.765916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:50.766050 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:50.766669 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:50.766674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:50.766818 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 15:53:50.766967 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 15:53:50.767269 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:53:50.767769 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.752470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.22.157.253:33815)
0928 15:53:50.753270 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:50.753277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:50.753306 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:53:50.753359 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:50.753365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:50.753664 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:50.753801 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:50.754893 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.755998 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.757903 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.758505 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.758690 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:50.762002 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:50.762018 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:50.762033 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:50.762062 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:50.764999 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:50.765665 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:50.765674 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:50.765679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:50.765798 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:50.766161 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:50.766168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:50.766171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:50.766541 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:50.766767 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:50.767134 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 15:53:50.767380 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":625,"thread_start_us":473,"threads_started":1}
I20260928 15:53:50.770535 4462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:53:50.771899 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.771355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33650)
0928 15:53:50.771642 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.771748 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 15:53:50.772226 4410 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
I20260928 15:53:50.773973 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.773544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33662)
0928 15:53:50.773706 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.773807 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:53:50.774238 4462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:45887: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab.
Entry for principal kudu/127.22.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab.
I20260928 15:53:50.784004 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.252:45887
--webserver_interface=127.22.157.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab
--principal=kudu/127.22.157.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:40313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:40711,127.22.157.253:33815,127.22.157.252:45887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:53:50.823586 4462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } attempt: 1
I20260928 15:53:50.824954 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.824348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33676)
0928 15:53:50.824621 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.824776 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 15:53:50.826122 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.825723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33686)
0928 15:53:50.825926 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.826002 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 15:53:50.826345 4462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:45887: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
I20260928 15:53:50.894620 4462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } attempt: 2
I20260928 15:53:50.895397 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } attempt: 5
I20260928 15:53:50.896138 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.895411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33690)
0928 15:53:50.895751 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.895893 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260928 15:53:50.897384 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.896995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33698)
0928 15:53:50.897145 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.897238 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 15:53:50.897681 4462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:45887: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Sep 28 15:53:50 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
I20260928 15:53:50.910581 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.896053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.22.157.253:33815)
0928 15:53:50.896315 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:50.896322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:50.896344 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:53:50.896571 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:50.896576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:50.896665 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:50.896826 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:50.898074 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.898786 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.900516 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.901044 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.901143 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:50.905058 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:50.905065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:50.905068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:50.905122 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:50.907851 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:50.908719 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:50.908765 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:50.908771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:50.908904 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:50.909332 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:50.909335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:50.909337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:50.909617 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:50.909723 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:50.910183 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 15:53:50.910357 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260928 15:53:50.910732 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.896040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46718)
0928 15:53:50.896234 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.896253 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:50.896383 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:50.896922 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:50.896929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:50.897578 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:50.897871 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:50.897887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.898975 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.898981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:50.900349 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:50.900365 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:50.901224 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:50.901230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:50.901426 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:50.902064 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:50.902086 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:50.904833 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:50.908028 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:50.908033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:50.908038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:50.908578 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:50.909021 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:50.909026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:50.909030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:50.909223 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:50.909744 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:50.909749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:50.909926 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 15:53:50.910079 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 15:53:50.910395 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 15:53:50.911953 4386 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:53:50.913311 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.912834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33708)
0928 15:53:50.913025 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.913134 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 15:53:50.914479 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.913986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33718)
0928 15:53:50.914206 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.914319 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 15:53:50.914731 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:45887: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
I20260928 15:53:50.966024 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } attempt: 1
I20260928 15:53:50.967451 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.966848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33722)
0928 15:53:50.967080 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.967207 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 15:53:50.968642 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.968180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33724)
0928 15:53:50.968376 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.968488 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 15:53:50.968969 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:45887: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
I20260928 15:53:50.988952 4462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } attempt: 3
I20260928 15:53:50.990262 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.989781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33736)
0928 15:53:50.989994 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.990099 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 15:53:50.991519 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:50.991020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33748)
0928 15:53:50.991299 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:50.991392 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
W20260928 15:53:50.991787 4462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:45887: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
I20260928 15:53:51.037230 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } attempt: 2
I20260928 15:53:51.038656 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.038020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33752)
0928 15:53:51.038292 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.038420 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 15:53:51.039812 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.039391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33764)
0928 15:53:51.039572 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.039672 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 15:53:51.040118 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:45887: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
W20260928 15:53:51.133709 4469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:51.134023 4469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:51.134114 4469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 15:53:51.135032 4462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } attempt: 4
I20260928 15:53:51.136283 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.135744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33778)
0928 15:53:51.136023 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.136126 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 15:53:51.137458 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.137104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33792)
0928 15:53:51.137259 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.137343 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:53:51.137714 4462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:45887: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
W20260928 15:53:51.142108 4469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:53:51.142231 4469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:51.142339 4469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:51.142455 4469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:53:51.142518 4469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:53:51.146353 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } attempt: 3
I20260928 15:53:51.147619 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.147058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33808)
0928 15:53:51.147315 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.147422 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 15:53:51.148708 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.148285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33818)
0928 15:53:51.148488 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.148569 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260928 15:53:51.148965 4386 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:45887: Network error: Client connection negotiation failed: client connection to 127.22.157.252:45887: connect: Connection refused (error 111)
I20260928 15:53:51.153733 4469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:40711,127.22.157.253:33815,127.22.157.252:45887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab
--principal=kudu/127.22.157.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.252:45887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.157.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:51.155158 4469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:51.156498 4469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:51.169039 4474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:51.168964 4475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:51.169066 4477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:51.170210 4469 server_base.cc:1034] running on GCE node
I20260928 15:53:51.171468 4469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:51.173214 4469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:51.174542 4469 hybrid_clock.cc:648] HybridClock initialized: now 1790610831174508 us; error 43 us; skew 500 ppm
I20260928 15:53:51.174981 4469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:51 dist-test-slave-1vw0 krb5kdc[4306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:51.178592 4469 init.cc:377] Logged in from keytab as kudu/127.22.157.252@KRBTEST.COM (short username kudu)
I20260928 15:53:51.180339 4469 webserver.cc:460] Webserver started at http://127.22.157.252:41383/ using document root <none> and password file <none>
I20260928 15:53:51.181143 4469 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:51.181254 4469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:51.181519 4469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:53:51.184715 4469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/data/instance:
uuid: "32121590944f43659d85325e822a9f64"
format_stamp: "Formatted at 2026-09-28 15:53:51 on dist-test-slave-1vw0"
server_key: "d83ce31dac147b9b348d8d509ae7b90e"
server_key_iv: "5317f11d7eccd1b40f9b10aca577f42d"
server_key_version: "encryptionkey@0"
I20260928 15:53:51.185832 4469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/wal/instance:
uuid: "32121590944f43659d85325e822a9f64"
format_stamp: "Formatted at 2026-09-28 15:53:51 on dist-test-slave-1vw0"
server_key: "d83ce31dac147b9b348d8d509ae7b90e"
server_key_iv: "5317f11d7eccd1b40f9b10aca577f42d"
server_key_version: "encryptionkey@0"
I20260928 15:53:51.191555 4469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:53:51.195415 4484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:51.197057 4469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 15:53:51.197264 4469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/wal
uuid: "32121590944f43659d85325e822a9f64"
format_stamp: "Formatted at 2026-09-28 15:53:51 on dist-test-slave-1vw0"
server_key: "d83ce31dac147b9b348d8d509ae7b90e"
server_key_iv: "5317f11d7eccd1b40f9b10aca577f42d"
server_key_version: "encryptionkey@0"
I20260928 15:53:51.197527 4469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:51.219785 4469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:51.224519 4469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:51.224872 4469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:51.244575 4469 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.252:45887
I20260928 15:53:51.244594 4536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.252:45887 every 8 connection(s)
I20260928 15:53:51.246126 4469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
I20260928 15:53:51.250144 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 4469
I20260928 15:53:51.250363 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790610618730370-23159-0/minicluster-data/master-2/wal/instance
I20260928 15:53:51.250455 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:51.250979 23159 external_mini_cluster.cc:1428] Setting key f216c937863e51b11ea7a77ab0cd9324
I20260928 15:53:51.260442 4537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:53:51 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
Sep 28 15:53:51 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:51.277626 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.252931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.22.157.252:45887)
0928 15:53:51.253814 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.253827 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.253876 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:53:51.255593 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.255617 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.256061 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.256434 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.258464 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.259788 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.261838 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.262861 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.263048 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.267909 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.267917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.267932 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.267985 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.271738 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.272873 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.272880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.272887 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.273112 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.273610 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.273614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.273616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.274530 (+ 914us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.274817 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.275522 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.276036 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":442,"threads_started":1}
I20260928 15:53:51.280783 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.263991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:40711 (local address 127.0.0.1:47820)
0928 15:53:51.264962 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.264999 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:51.265190 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:51.265587 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:51.265594 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:51.266254 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:51.266517 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.266526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.267668 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.267674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.269077 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.269094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.269913 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.269920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.270131 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.270806 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:51.270832 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:51.273551 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:51.277892 (+ 4341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.277902 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.277908 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.278495 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.278914 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.278919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.278922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.279089 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.279761 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:51.279766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:51.279916 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 15:53:51.280088 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 15:53:51.280436 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":544,"threads_started":1}
I20260928 15:53:51.281036 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.263990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.22.157.254:40711)
0928 15:53:51.264624 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.264631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.265099 (+ 468us) server_negotiation.cc:407] Connection header received
0928 15:53:51.265321 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.265331 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.265397 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.265516 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.266690 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.267501 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.269251 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.269698 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.269841 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.273806 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.273813 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.273818 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.273864 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.277716 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.278643 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.278648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.278650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.278791 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.279222 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.279228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.279234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.279629 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.279794 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.280248 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.280561 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":300,"threads_started":1}
I20260928 15:53:51.282384 4537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:53:51 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
Sep 28 15:53:51 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:51.297065 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.283216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:33815 (local address 127.0.0.1:46732)
0928 15:53:51.283386 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.283400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:51.283512 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:51.283837 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:51.283842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:51.284201 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:51.284339 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.284347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.285343 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.285350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.286223 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.286234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.287137 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.287143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.287385 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.287874 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:51.287900 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:51.290495 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:51.293889 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.293897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.293901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.294494 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.294935 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.294942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.294949 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.295136 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.295958 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:51.295963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:51.296183 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 15:53:51.296348 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 15:53:51.296595 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 15:53:51.297097 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.283143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.22.157.253:33815)
0928 15:53:51.283334 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.283340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.283466 (+ 126us) server_negotiation.cc:407] Connection header received
0928 15:53:51.283641 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.283647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.283716 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.283815 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.284478 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.285213 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.286366 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.286889 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.287022 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.290720 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.290730 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.290738 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.290787 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.293733 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.294652 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.294656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.294659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.294808 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.295306 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.295311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.295314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.295755 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.295946 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.296501 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.296837 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 15:53:51.298240 4537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:53:51.298555 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.281465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.22.157.254:40711)
0928 15:53:51.281860 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.281864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.281877 (+ 13us) server_negotiation.cc:407] Connection header received
0928 15:53:51.281911 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.281914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.281978 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.282085 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.283634 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.284317 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.285671 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.286078 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.286206 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.291658 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.291667 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.291675 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.291734 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.294889 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.295951 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.295958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.295961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.296146 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.296678 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.296682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.296686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.297082 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.297370 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.298002 (+ 632us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.298273 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
Sep 28 15:53:51 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610831, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
Sep 28 15:53:51 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
I20260928 15:53:51.311932 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.299122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.22.157.252:45887)
0928 15:53:51.299319 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.299324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.299542 (+ 218us) server_negotiation.cc:407] Connection header received
0928 15:53:51.299620 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.299625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.299705 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.299812 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.300668 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.301505 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.302869 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.303286 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.303420 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.306546 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.306551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.306554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.306581 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.309418 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.310049 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.310056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.310059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.310181 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.310524 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.310532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.310535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.310823 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.310974 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.311316 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.311551 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:53:51.312055 4542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.299217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33832)
0928 15:53:51.299365 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.299387 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:51.299504 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:51.299827 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:51.299832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:51.300338 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:51.300527 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.300541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.301663 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.301668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.302742 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.302754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.303436 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.303441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.303649 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.304259 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:51.304275 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:51.306372 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:51.309551 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.309557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.309561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.309932 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.310285 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.310291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.310295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.310424 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.310935 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:51.310939 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:51.311091 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 15:53:51.311242 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 15:53:51.311717 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":148}
I20260928 15:53:51.316782 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.300320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.22.157.253:33815)
0928 15:53:51.300463 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.300467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.300509 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:53:51.300780 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.300785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.300858 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.300961 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.302462 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.303260 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.304802 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.305222 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.305377 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.310212 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.310218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.310221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.310258 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.313829 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.314643 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.314650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.314654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.314809 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.315241 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.315246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.315249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.315549 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.315845 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.316153 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.316400 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:53:51.317876 4537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64: Bootstrap starting.
I20260928 15:53:51.322188 4537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:51.323493 4537 log.cc:824] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:51.323684 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:53:51.326195 4386 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } attempt: 4
I20260928 15:53:51.327411 4537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64: No bootstrap required, opened a new log
Sep 28 15:53:51 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
I20260928 15:53:51.342849 4537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.348783 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.326831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33842)
0928 15:53:51.327020 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.327042 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:51.327172 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:51.327740 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:51.327750 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:51.328454 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:51.328782 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.328800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.330543 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.330550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.332106 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.332131 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.333802 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.333811 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.334124 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.335056 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:51.335092 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:51.339094 (+ 4002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:51.345251 (+ 6157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.345261 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.345268 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.346039 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.346680 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.346688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.346694 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.346907 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.347610 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:51.347615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:51.347812 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 15:53:51.347958 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 15:53:51.348288 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 15:53:51.349027 4537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:51.350417 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.326875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.22.157.252:45887)
0928 15:53:51.327341 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.327347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.327367 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:53:51.327441 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.327445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.327524 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.327677 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.329007 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.330316 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.332393 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.333559 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.339377 (+ 5818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.339761 (+ 384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.339772 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.339780 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.339827 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.345015 (+ 5188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.346270 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.346279 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.346285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.346517 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.347080 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.347084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.347087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.347473 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.349600 (+ 2127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.349750 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.350000 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 15:53:51.351326 4537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32121590944f43659d85325e822a9f64, State: Initialized, Role: FOLLOWER
I20260928 15:53:51.353039 4537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.359026 4386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e: Bootstrap starting.
I20260928 15:53:51.361800 4541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.331348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.22.157.254:40711)
0928 15:53:51.331532 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.331537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.337480 (+ 5943us) server_negotiation.cc:407] Connection header received
0928 15:53:51.343618 (+ 6138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.343624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.343721 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.343799 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.346106 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.347177 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.349725 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.350339 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.350433 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.354252 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.354259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.354264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.354305 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.358175 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.359170 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.359178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.359182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.359335 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.359819 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.359824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.359827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.360208 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.360461 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.361097 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.361392 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:53:51.363679 4386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:51.366210 4386 log.cc:824] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:51.370625 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.331841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.22.157.252:45887)
0928 15:53:51.347272 (+ 15431us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.347277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.347299 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:53:51.347347 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.347352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.347420 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.350113 (+ 2693us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.350395 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.351652 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.354124 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.355116 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.359473 (+ 4357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.359818 (+ 345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.359828 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.359836 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.359878 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.365879 (+ 6001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.367087 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.367095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.367098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.367353 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.367919 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.367927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.367933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.368479 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.369841 (+ 1362us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.369981 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.370212 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15280,"thread_start_us":15687,"threads_started":1}
I20260928 15:53:51.371455 4537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:53:51.369920 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.372200 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:53:51.374176 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.338351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.22.157.253:33815)
0928 15:53:51.339027 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.339033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.342225 (+ 3192us) server_negotiation.cc:407] Connection header received
0928 15:53:51.342773 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.342780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.342878 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.342959 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.345616 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.346801 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.349184 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.349855 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.349959 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.365607 (+ 15648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.365616 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.365621 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.365667 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.370142 (+ 4475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.371447 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.371457 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.371463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.371746 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.372263 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.372271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.372275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.372688 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.372972 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.373533 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.373795 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526}
I20260928 15:53:51.374660 4386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e: No bootstrap required, opened a new log
W20260928 15:53:51.386420 4564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:53:51.386683 4564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:53:51.387442 4386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.387823 4386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:51.387974 4386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f397d35a984790bb467c27f19ab94e, State: Initialized, Role: FOLLOWER
I20260928 15:53:51.388686 4386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.389736 4566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.390209 4566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:53:51.391000 4386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:53:51.383206 4469 master_runner.cc:186] Error getting master registration for 127.22.157.254:40711: OK, instance_id { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" instance_seqno: 1790610830176981 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:53:51.396533 4469 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 15:53:51.402951 4577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:53:51.403221 4577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:53:51.402209 4319 master_runner.cc:186] Error getting master registration for 127.22.157.253:33815: OK, instance_id { permanent_uuid: "c264fe1c6af646648913367a481adcf4" instance_seqno: 1790610830701575 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:53:51.413527 4319 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 15:53:51.415335 4340 tablet.cc:2369] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:53:51.436962 4462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } attempt: 5
Sep 28 15:53:51 dist-test-slave-1vw0 krb5kdc[4306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610830, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
I20260928 15:53:51.456344 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.437645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:45887 (local address 127.0.0.1:33866)
0928 15:53:51.437943 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:51.437980 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:51.438105 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:51.438502 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:51.438507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:51.439083 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:51.439307 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.439320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.440861 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.440866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.442347 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:51.442365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.443443 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.443450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:51.443692 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.444502 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:51.444556 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:51.448203 (+ 3647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:51.453068 (+ 4865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.453075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.453082 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.453680 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.454210 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:51.454216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:51.454220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:51.454444 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:51.455194 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:51.455200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:51.455387 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 15:53:51.455565 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 15:53:51.455898 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260928 15:53:51.456377 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:51.437717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.22.157.252:45887)
0928 15:53:51.437967 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:51.437974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:51.438034 (+ 60us) server_negotiation.cc:407] Connection header received
0928 15:53:51.438241 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:51.438249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:51.438331 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:51.438451 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:51.439507 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.440658 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.442529 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:51.443253 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:51.443382 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:51.448522 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:51.448530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:51.448535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:51.448579 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:51.452831 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.453844 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.453851 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.453855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.454047 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:51.454579 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:51.454584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:51.454588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:51.454970 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:51.455140 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:51.455713 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 15:53:51.455970 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:53:51.464833 4462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4: Bootstrap starting.
I20260928 15:53:51.469713 4462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:51.471283 4462 log.cc:824] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:51.476168 4462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4: No bootstrap required, opened a new log
W20260928 15:53:51.477312 4490 tablet.cc:2369] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:53:51.488202 4462 raft_consensus.cc:348] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.488747 4462 raft_consensus.cc:374] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:51.488879 4462 raft_consensus.cc:729] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c264fe1c6af646648913367a481adcf4, State: Initialized, Role: FOLLOWER
I20260928 15:53:51.489692 4462 consensus_queue.cc:260] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.490787 4580 sys_catalog.cc:455] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.491340 4580 sys_catalog.cc:458] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:53:51.492307 4462 sys_catalog.cc:564] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 15:53:51.508853 4591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:53:51.509120 4591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:53:51.516140 4566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:53:51.516453 4566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.516701 4393 master_runner.cc:263] No leader master found from master c264fe1c6af646648913367a481adcf4
I20260928 15:53:51.516887 4393 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:53:51.518155 4566 leader_election.cc:288] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c264fe1c6af646648913367a481adcf4 (127.22.157.253:33815), 32121590944f43659d85325e822a9f64 (127.22.157.252:45887)
I20260928 15:53:51.518870 4552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:53:51.519225 4552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.519258 4511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41f397d35a984790bb467c27f19ab94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32121590944f43659d85325e822a9f64" is_pre_election: true
I20260928 15:53:51.519891 4511 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41f397d35a984790bb467c27f19ab94e in term 0.
I20260928 15:53:51.520563 4336 leader_election.cc:302] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32121590944f43659d85325e822a9f64, 41f397d35a984790bb467c27f19ab94e; no voters:
I20260928 15:53:51.520808 4552 leader_election.cc:288] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41f397d35a984790bb467c27f19ab94e (127.22.157.254:40711), c264fe1c6af646648913367a481adcf4 (127.22.157.253:33815)
I20260928 15:53:51.521150 4566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:53:51.521314 4566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:53:51.521409 4566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:51.522399 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32121590944f43659d85325e822a9f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264fe1c6af646648913367a481adcf4" is_pre_election: true
I20260928 15:53:51.522423 4436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41f397d35a984790bb467c27f19ab94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264fe1c6af646648913367a481adcf4" is_pre_election: true
I20260928 15:53:51.523192 4435 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32121590944f43659d85325e822a9f64 in term 0.
I20260928 15:53:51.523825 4488 leader_election.cc:302] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32121590944f43659d85325e822a9f64, c264fe1c6af646648913367a481adcf4; no voters:
I20260928 15:53:51.524348 4552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:53:51.524492 4552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:53:51.524601 4552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:51.525454 4566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.526047 4361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32121590944f43659d85325e822a9f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f397d35a984790bb467c27f19ab94e" is_pre_election: true
I20260928 15:53:51.527177 4361 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 32121590944f43659d85325e822a9f64 in current term 1: Already voted for candidate 41f397d35a984790bb467c27f19ab94e in this term.
I20260928 15:53:51.528420 4552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.529337 4511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41f397d35a984790bb467c27f19ab94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32121590944f43659d85325e822a9f64"
I20260928 15:53:51.529346 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41f397d35a984790bb467c27f19ab94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264fe1c6af646648913367a481adcf4"
I20260928 15:53:51.529610 4511 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 41f397d35a984790bb467c27f19ab94e in current term 1: Already voted for candidate 32121590944f43659d85325e822a9f64 in this term.
I20260928 15:53:51.529661 4435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:51.529660 4361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32121590944f43659d85325e822a9f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f397d35a984790bb467c27f19ab94e"
I20260928 15:53:51.529925 4361 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 32121590944f43659d85325e822a9f64 in current term 1: Already voted for candidate 41f397d35a984790bb467c27f19ab94e in this term.
I20260928 15:53:51.530360 4566 leader_election.cc:288] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [CANDIDATE]: Term 1 election: Requested vote from peers c264fe1c6af646648913367a481adcf4 (127.22.157.253:33815), 32121590944f43659d85325e822a9f64 (127.22.157.252:45887)
I20260928 15:53:51.533396 4435 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41f397d35a984790bb467c27f19ab94e in term 1.
I20260928 15:53:51.533977 4338 leader_election.cc:302] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41f397d35a984790bb467c27f19ab94e, c264fe1c6af646648913367a481adcf4; no voters: 32121590944f43659d85325e822a9f64
I20260928 15:53:51.534608 4566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:53:51.536912 4435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32121590944f43659d85325e822a9f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c264fe1c6af646648913367a481adcf4"
I20260928 15:53:51.537326 4435 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 32121590944f43659d85325e822a9f64 in current term 1: Already voted for candidate 41f397d35a984790bb467c27f19ab94e in this term.
I20260928 15:53:51.537994 4488 leader_election.cc:302] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32121590944f43659d85325e822a9f64; no voters: 41f397d35a984790bb467c27f19ab94e, c264fe1c6af646648913367a481adcf4
I20260928 15:53:51.539301 4552 leader_election.cc:288] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [CANDIDATE]: Term 1 election: Requested vote from peers 41f397d35a984790bb467c27f19ab94e (127.22.157.254:40711), c264fe1c6af646648913367a481adcf4 (127.22.157.253:33815)
I20260928 15:53:51.540164 4552 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 15:53:51.546615 4566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [term 1 LEADER]: Becoming Leader. State: Replica: 41f397d35a984790bb467c27f19ab94e, State: Running, Role: LEADER
I20260928 15:53:51.547358 4566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } }
I20260928 15:53:51.566126 4593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f397d35a984790bb467c27f19ab94e. Latest consensus state: current_term: 1 leader_uuid: "41f397d35a984790bb467c27f19ab94e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.566625 4593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:51.568039 4597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:53:51.571298 4597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:53:51.596626 4511 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [term 1 FOLLOWER]: Refusing update from remote peer 41f397d35a984790bb467c27f19ab94e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:53:51.597711 4566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:53:51.605839 4435 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [term 1 FOLLOWER]: Refusing update from remote peer 41f397d35a984790bb467c27f19ab94e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:53:51.607093 4566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:53:51.644294 4580 sys_catalog.cc:455] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f397d35a984790bb467c27f19ab94e. Latest consensus state: current_term: 1 leader_uuid: "41f397d35a984790bb467c27f19ab94e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.644649 4580 sys_catalog.cc:458] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:53:51.648598 4598 mvcc.cc:205] Tried to move back new op lower bound from 7334341966206267392 to 7334341966022680576. Current Snapshot: MvccSnapshot[applied={T|T < 7334341966206267392}]
I20260928 15:53:51.653362 4580 sys_catalog.cc:455] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41f397d35a984790bb467c27f19ab94e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.653707 4580 sys_catalog.cc:458] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:53:51.654971 4600 mvcc.cc:205] Tried to move back new op lower bound from 7334341966206267392 to 7334341966022680576. Current Snapshot: MvccSnapshot[applied={T|T < 7334341966206267392}]
I20260928 15:53:51.656713 4594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f397d35a984790bb467c27f19ab94e. Latest consensus state: current_term: 1 leader_uuid: "41f397d35a984790bb467c27f19ab94e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.657243 4594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:53:51.665337 4599 mvcc.cc:205] Tried to move back new op lower bound from 7334341966206267392 to 7334341966022680576. Current Snapshot: MvccSnapshot[applied={T|T < 7334341966206267392}]
I20260928 15:53:51.665310 4566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41f397d35a984790bb467c27f19ab94e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.665725 4566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:51.666056 4597 catalog_manager.cc:1347] Generated new cluster ID: 0d3721ed08a14ad683a6e34aa7fefdd6
I20260928 15:53:51.666167 4597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:53:51.670493 4594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41f397d35a984790bb467c27f19ab94e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.670845 4594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:53:51.676062 4566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41f397d35a984790bb467c27f19ab94e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f397d35a984790bb467c27f19ab94e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40711 } } peers { permanent_uuid: "c264fe1c6af646648913367a481adcf4" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 33815 } } peers { permanent_uuid: "32121590944f43659d85325e822a9f64" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 45887 } } }
I20260928 15:53:51.676445 4566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:51.703059 4597 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:53:51.705191 4597 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:53:51.723959 4597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f397d35a984790bb467c27f19ab94e: Generated new TSK 0
I20260928 15:53:51.725198 4597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:53:52.388787 4564 catalog_manager.cc:1259] Loaded cluster ID: 0d3721ed08a14ad683a6e34aa7fefdd6
I20260928 15:53:52.388890 4564 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64: loading cluster ID for follower catalog manager: success
I20260928 15:53:52.391482 4564 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64: acquiring CA information for follower catalog manager: success
I20260928 15:53:52.393498 4564 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 32121590944f43659d85325e822a9f64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:53:52.511234 4591 catalog_manager.cc:1259] Loaded cluster ID: 0d3721ed08a14ad683a6e34aa7fefdd6
I20260928 15:53:52.511343 4591 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4: loading cluster ID for follower catalog manager: success
I20260928 15:53:52.513836 4591 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4: acquiring CA information for follower catalog manager: success
I20260928 15:53:52.515694 4591 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c264fe1c6af646648913367a481adcf4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:53:52.885589 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 4319
I20260928 15:53:52.887140 4381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:53.003580 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 4319
I20260928 15:53:53.026805 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 4393
I20260928 15:53:53.028381 4457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:53.137935 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 4393
I20260928 15:53:53.159248 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 4469
I20260928 15:53:53.160548 4532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:53.272359 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 4469
2026-09-28T15:53:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5002 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:53:53 dist-test-slave-1vw0 krb5kdc[4616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:53:53 dist-test-slave-1vw0 krb5kdc[4616](info): set up 2 sockets
Sep 28 15:53:53 dist-test-slave-1vw0 krb5kdc[4616](info): commencing operation
krb5kdc: starting...
W20260928 15:53:54.405761 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:53:54 dist-test-slave-1vw0 krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:53:54.676951 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42969
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:42969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:53:55.027563 4629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:55.027895 4629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:55.027987 4629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:55.035384 4629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:53:55.035522 4629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:55.035578 4629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:55.035630 4629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:53:55.035677 4629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:53:55.045965 4629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:42969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:42969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:55.047318 4629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:55.048637 4629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:55.061707 4634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:55.061642 4635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:55.061668 4637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:55.062908 4629 server_base.cc:1034] running on GCE node
I20260928 15:53:55.063773 4629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:55.065397 4629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:55.066605 4629 hybrid_clock.cc:648] HybridClock initialized: now 1790610835066563 us; error 34 us; skew 500 ppm
I20260928 15:53:55.066936 4629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:55 dist-test-slave-1vw0 krb5kdc[4616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610835, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:55.085006 4629 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:53:55.086788 4629 webserver.cc:460] Webserver started at http://127.22.157.254:34895/ using document root <none> and password file <none>
I20260928 15:53:55.087594 4629 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:55.087764 4629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:55.088112 4629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:53:55.091557 4629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "cdd897807d25489ebb0c570623b2d8f8"
format_stamp: "Formatted at 2026-09-28 15:53:55 on dist-test-slave-1vw0"
server_key: "1ea11212b32d30adb6dfa1c33810e03e"
server_key_iv: "fe33aa36efa0de45527d789430d800c2"
server_key_version: "encryptionkey@0"
I20260928 15:53:55.092427 4629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "cdd897807d25489ebb0c570623b2d8f8"
format_stamp: "Formatted at 2026-09-28 15:53:55 on dist-test-slave-1vw0"
server_key: "1ea11212b32d30adb6dfa1c33810e03e"
server_key_iv: "fe33aa36efa0de45527d789430d800c2"
server_key_version: "encryptionkey@0"
I20260928 15:53:55.098292 4629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:53:55.102375 4644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:55.103957 4629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:53:55.104176 4629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "cdd897807d25489ebb0c570623b2d8f8"
format_stamp: "Formatted at 2026-09-28 15:53:55 on dist-test-slave-1vw0"
server_key: "1ea11212b32d30adb6dfa1c33810e03e"
server_key_iv: "fe33aa36efa0de45527d789430d800c2"
server_key_version: "encryptionkey@0"
I20260928 15:53:55.104427 4629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:55.131695 4629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:55.136814 4629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:55.137166 4629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:55.157186 4629 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:42969
I20260928 15:53:55.157209 4695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:42969 every 8 connection(s)
I20260928 15:53:55.158762 4629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:53:55.162740 4696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:55.164587 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 4629
I20260928 15:53:55.164870 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:53:55.165750 23159 external_mini_cluster.cc:1428] Setting key 348b383899071a879cf58be9123aca14
I20260928 15:53:55.177138 4696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8: Bootstrap starting.
Sep 28 15:53:55 dist-test-slave-1vw0 krb5kdc[4616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:55.180822 4696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:55.181953 4696 log.cc:824] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:55.185555 4696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8: No bootstrap required, opened a new log
I20260928 15:53:55.190544 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:55.167950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.22.157.254:42969)
0928 15:53:55.168881 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:55.168895 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:55.168947 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:53:55.170640 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:55.170668 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:55.171117 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:55.171544 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:55.173292 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:55.174740 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:55.176187 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:55.177035 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:55.177268 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:55.182040 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:55.182049 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:55.182067 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:55.182143 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:55.185810 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:55.186722 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:55.186736 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:55.186743 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:55.186919 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:55.187372 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:55.187378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:55.187386 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:55.187988 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:55.188348 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:55.188879 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 15:53:55.189379 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":372,"threads_started":1}
I20260928 15:53:55.194761 4696 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd897807d25489ebb0c570623b2d8f8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42969 } }
I20260928 15:53:55.195290 4696 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:55.195323 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:53:55.195405 4696 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd897807d25489ebb0c570623b2d8f8, State: Initialized, Role: FOLLOWER
I20260928 15:53:55.196043 4696 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd897807d25489ebb0c570623b2d8f8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42969 } }
I20260928 15:53:55.196295 4696 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:53:55.196449 4696 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:53:55.196630 4696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:55.199223 4696 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd897807d25489ebb0c570623b2d8f8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42969 } }
I20260928 15:53:55.199756 4696 leader_election.cc:302] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd897807d25489ebb0c570623b2d8f8; no voters:
I20260928 15:53:55.200361 4696 leader_election.cc:288] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:53:55.200567 4701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:53:55.201321 4701 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [term 1 LEADER]: Becoming Leader. State: Replica: cdd897807d25489ebb0c570623b2d8f8, State: Running, Role: LEADER
I20260928 15:53:55.201932 4701 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd897807d25489ebb0c570623b2d8f8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42969 } }
I20260928 15:53:55.202641 4696 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:53:55.205265 4703 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd897807d25489ebb0c570623b2d8f8. Latest consensus state: current_term: 1 leader_uuid: "cdd897807d25489ebb0c570623b2d8f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd897807d25489ebb0c570623b2d8f8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42969 } } }
I20260928 15:53:55.205782 4703 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:55.206115 4702 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd897807d25489ebb0c570623b2d8f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd897807d25489ebb0c570623b2d8f8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 42969 } } }
I20260928 15:53:55.206601 4702 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:55.209268 4706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:53:55.215246 4706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:53:55.226274 4706 catalog_manager.cc:1347] Generated new cluster ID: ba118d41752d43c49d53a959d7b4f465
I20260928 15:53:55.226456 4706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:53:55.241256 4706 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:53:55.242699 4706 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:53:55.252499 4706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd897807d25489ebb0c570623b2d8f8: Generated new TSK 0
I20260928 15:53:55.253582 4706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:53:55.309082 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 4629
I20260928 15:53:55.310581 4691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:55.411353 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 4629
2026-09-28T15:53:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2132 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:53:55 dist-test-slave-1vw0 krb5kdc[4725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:53:55 dist-test-slave-1vw0 krb5kdc[4725](info): set up 2 sockets
Sep 28 15:53:55 dist-test-slave-1vw0 krb5kdc[4725](info): commencing operation
krb5kdc: starting...
W20260928 15:53:56.546792 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:53:56 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:53:56.838145 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:32997
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:53:57.219648 4738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:57.220070 4738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:57.220239 4738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:57.228344 4738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:53:57.228549 4738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:57.228667 4738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:57.228803 4738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:53:57.228891 4738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:53:57.239934 4738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:57.241655 4738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:57.243127 4738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:57.256217 4743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:57.256196 4744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:57.256173 4746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:57.257102 4738 server_base.cc:1034] running on GCE node
I20260928 15:53:57.257973 4738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:57.259557 4738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:57.260861 4738 hybrid_clock.cc:648] HybridClock initialized: now 1790610837260797 us; error 46 us; skew 500 ppm
I20260928 15:53:57.261257 4738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:57 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610837, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:57.264709 4738 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:53:57.266587 4738 webserver.cc:460] Webserver started at http://127.22.157.254:37055/ using document root <none> and password file <none>
I20260928 15:53:57.267287 4738 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:57.267395 4738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:57.267666 4738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:53:57.271104 4738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "af760abfe67d4c08be7f8d49227da4b8"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "f7d6c0fe1da714ccb880d0696cbc42a5"
server_key_iv: "69ac1a4fe4b7acf45713fd16f85c2bf2"
server_key_version: "encryptionkey@0"
I20260928 15:53:57.271925 4738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "af760abfe67d4c08be7f8d49227da4b8"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "f7d6c0fe1da714ccb880d0696cbc42a5"
server_key_iv: "69ac1a4fe4b7acf45713fd16f85c2bf2"
server_key_version: "encryptionkey@0"
I20260928 15:53:57.277850 4738 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 15:53:57.281961 4753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:57.283663 4738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:53:57.283890 4738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "af760abfe67d4c08be7f8d49227da4b8"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "f7d6c0fe1da714ccb880d0696cbc42a5"
server_key_iv: "69ac1a4fe4b7acf45713fd16f85c2bf2"
server_key_version: "encryptionkey@0"
I20260928 15:53:57.284102 4738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:57.300231 4738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:57.305225 4738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:57.305557 4738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:57.327147 4738 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:32997
I20260928 15:53:57.327179 4804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:32997 every 8 connection(s)
I20260928 15:53:57.328680 4738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:53:57.332666 4805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:57.337005 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 4738
I20260928 15:53:57.337249 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:53:57.337955 23159 external_mini_cluster.cc:1428] Setting key ddfcead4378d3ee692aafa434696688f
I20260928 15:53:57.344700 4805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Bootstrap starting.
I20260928 15:53:57.348595 4805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:57.349912 4805 log.cc:824] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:57.354058 4805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: No bootstrap required, opened a new log
Sep 28 15:53:57 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:57.362474 4805 raft_consensus.cc:348] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:53:57.362918 4805 raft_consensus.cc:374] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:57.363059 4805 raft_consensus.cc:729] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af760abfe67d4c08be7f8d49227da4b8, State: Initialized, Role: FOLLOWER
I20260928 15:53:57.363682 4805 consensus_queue.cc:260] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:53:57.363940 4805 raft_consensus.cc:388] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:53:57.364112 4805 raft_consensus.cc:482] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:53:57.364307 4805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:57.364678 4808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:57.340183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.22.157.254:32997)
0928 15:53:57.341157 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:57.341169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:57.341220 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:53:57.343107 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:57.343134 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:57.343582 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:57.343945 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:57.345990 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:57.347413 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:57.349328 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:57.350234 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:57.350378 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:57.356052 (+ 5674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:57.356061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:57.356082 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:57.356126 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:57.359587 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:57.360453 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:57.360464 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:57.360472 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:57.360681 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:57.361180 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:57.361189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:57.361192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:57.361832 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:57.362158 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:57.362669 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 15:53:57.363375 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":409,"threads_started":1}
I20260928 15:53:57.367349 4805 raft_consensus.cc:504] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:53:57.367959 4805 leader_election.cc:302] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af760abfe67d4c08be7f8d49227da4b8; no voters:
I20260928 15:53:57.368626 4805 leader_election.cc:288] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:53:57.368783 4810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:53:57.370216 4810 raft_consensus.cc:686] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 1 LEADER]: Becoming Leader. State: Replica: af760abfe67d4c08be7f8d49227da4b8, State: Running, Role: LEADER
I20260928 15:53:57.370850 4810 consensus_queue.cc:237] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:53:57.371115 4805 sys_catalog.cc:564] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:53:57.374042 4812 sys_catalog.cc:455] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af760abfe67d4c08be7f8d49227da4b8. Latest consensus state: current_term: 1 leader_uuid: "af760abfe67d4c08be7f8d49227da4b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } } }
I20260928 15:53:57.374116 4811 sys_catalog.cc:455] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af760abfe67d4c08be7f8d49227da4b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } } }
I20260928 15:53:57.374557 4811 sys_catalog.cc:458] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:57.374439 4812 sys_catalog.cc:458] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:53:57.377667 4818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:53:57.384580 4818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:53:57.397347 4818 catalog_manager.cc:1347] Generated new cluster ID: 84b100c5970d4d31a879665cd606f395
I20260928 15:53:57.397485 4818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:53:57.421010 4818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:53:57.422726 4818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:53:57.433655 4818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Generated new TSK 0
I20260928 15:53:57.434895 4818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:53:57.509703 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:32997
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:53:57.883978 4831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:57.884359 4831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:57.884547 4831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:57.892695 4831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:57.892940 4831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:57.893146 4831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:53:57.903640 4831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:57.905191 4831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:57.906664 4831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:57.921109 4837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:57.921120 4836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:57.921283 4839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:57.921784 4831 server_base.cc:1034] running on GCE node
I20260928 15:53:57.922482 4831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:57.923600 4831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:57.924880 4831 hybrid_clock.cc:648] HybridClock initialized: now 1790610837924814 us; error 46 us; skew 500 ppm
I20260928 15:53:57.925318 4831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:57 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610837, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:57.928933 4831 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:53:57.930896 4831 webserver.cc:460] Webserver started at http://127.22.157.193:33667/ using document root <none> and password file <none>
I20260928 15:53:57.931674 4831 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:57.931789 4831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:57.932114 4831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:53:57.935690 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "8151cbdb27444d0e94e304ff726a159a"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "2b7fed6f9eab1a40e7fab32e40eee98a"
server_key_iv: "b36e1f7f6dcff3e56626e576cd606553"
server_key_version: "encryptionkey@0"
I20260928 15:53:57.936609 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "8151cbdb27444d0e94e304ff726a159a"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "2b7fed6f9eab1a40e7fab32e40eee98a"
server_key_iv: "b36e1f7f6dcff3e56626e576cd606553"
server_key_version: "encryptionkey@0"
I20260928 15:53:57.942451 4831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:53:57.946345 4846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:57.948063 4831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:53:57.948276 4831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "8151cbdb27444d0e94e304ff726a159a"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "2b7fed6f9eab1a40e7fab32e40eee98a"
server_key_iv: "b36e1f7f6dcff3e56626e576cd606553"
server_key_version: "encryptionkey@0"
I20260928 15:53:57.948524 4831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:57.969615 4831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:57.975003 4831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:57.975325 4831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:57.976434 4831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:53:57.977905 4831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:53:57.977993 4831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:57.978089 4831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:53:57.978133 4831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:58.020660 4831 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:35657
I20260928 15:53:58.020682 4959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:35657 every 8 connection(s)
I20260928 15:53:58.022121 4831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:53:58.030917 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 4831
I20260928 15:53:58.031164 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:53:58.032019 23159 external_mini_cluster.cc:1428] Setting key 0155c745b481306acdd099046ac4c3a0
Sep 28 15:53:58 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610837, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:58.046988 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:58.025018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:56527 (local address 127.22.157.254:32997)
0928 15:53:58.025467 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:58.025472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:58.026647 (+ 1175us) server_negotiation.cc:407] Connection header received
0928 15:53:58.028564 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:58.028568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:58.028646 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:58.028812 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:58.030581 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:58.031818 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:58.034013 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:58.034646 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:58.034799 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:58.039085 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:58.039095 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:58.039103 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:58.039155 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:58.043019 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:58.044039 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:58.044049 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:58.044056 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:58.044240 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:58.044935 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:58.044939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:58.044947 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:58.045312 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:58.045448 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:58.046503 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0928 15:53:58.046714 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":217,"threads_started":1}
I20260928 15:53:58.048763 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:58.025658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:56527)
0928 15:53:58.026440 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:58.026499 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:58.028261 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:58.029093 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:58.029103 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:58.029670 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:58.030332 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:58.030353 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:58.032065 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:58.032069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:58.033783 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:58.033800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:58.034917 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:58.034923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:58.035166 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:58.035943 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:58.035976 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:58.038774 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:58.043261 (+ 4487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:58.043270 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:58.043286 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:58.043861 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:58.044407 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:58.044414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:58.044419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:58.044760 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:58.045464 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:58.045476 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:58.046007 (+ 531us) client_negotiation.cc:769] Sending connection context
0928 15:53:58.046384 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 15:53:58.047059 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":306,"threads_started":1}
I20260928 15:53:58.050678 4960 heartbeater.cc:347] Connected to a master server at 127.22.157.254:32997
I20260928 15:53:58.051029 4960 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:58.051975 4960 heartbeater.cc:511] Master 127.22.157.254:32997 requested a full tablet report, sending...
I20260928 15:53:58.054769 4770 ts_manager.cc:194] Registered new tserver with Master: 8151cbdb27444d0e94e304ff726a159a (127.22.157.193:35657)
I20260928 15:53:58.057185 4770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:56527
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:53:58.097074 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:32997
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:53:58.441834 4966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:58.442140 4966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:58.442310 4966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:58.449924 4966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:58.450132 4966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:58.450327 4966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:53:58.460876 4966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:58.462391 4966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:58.463802 4966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:58.478099 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:58.478073 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:58.478035 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:58.478960 4966 server_base.cc:1034] running on GCE node
I20260928 15:53:58.479665 4966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:58.480922 4966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:58.482232 4966 hybrid_clock.cc:648] HybridClock initialized: now 1790610838482164 us; error 66 us; skew 500 ppm
I20260928 15:53:58.482610 4966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:58 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610838, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:58.486085 4966 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:53:58.487735 4966 webserver.cc:460] Webserver started at http://127.22.157.194:45581/ using document root <none> and password file <none>
I20260928 15:53:58.488508 4966 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:58.488658 4966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:58.489008 4966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:53:58.492108 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "2a7ba31dcac04400b85f8642f750e569"
format_stamp: "Formatted at 2026-09-28 15:53:58 on dist-test-slave-1vw0"
server_key: "1e2f020bc482d54a468a3103b43027f5"
server_key_iv: "df101ceb49efb26f30170c080afe5bf3"
server_key_version: "encryptionkey@0"
I20260928 15:53:58.493063 4966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "2a7ba31dcac04400b85f8642f750e569"
format_stamp: "Formatted at 2026-09-28 15:53:58 on dist-test-slave-1vw0"
server_key: "1e2f020bc482d54a468a3103b43027f5"
server_key_iv: "df101ceb49efb26f30170c080afe5bf3"
server_key_version: "encryptionkey@0"
I20260928 15:53:58.498657 4966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 15:53:58.502794 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:58.504350 4966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:53:58.504567 4966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "2a7ba31dcac04400b85f8642f750e569"
format_stamp: "Formatted at 2026-09-28 15:53:58 on dist-test-slave-1vw0"
server_key: "1e2f020bc482d54a468a3103b43027f5"
server_key_iv: "df101ceb49efb26f30170c080afe5bf3"
server_key_version: "encryptionkey@0"
I20260928 15:53:58.504881 4966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:58.540035 4966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:58.545042 4966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:58.545368 4966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:58.546579 4966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:53:58.554105 4966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:53:58.554306 4966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 15:53:58.554488 4966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:53:58.554564 4966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:58.591318 4966 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:35389
I20260928 15:53:58.591373 5094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:35389 every 8 connection(s)
I20260928 15:53:58.592741 4966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:53:58.595301 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 4966
I20260928 15:53:58.595494 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:53:58.596129 23159 external_mini_cluster.cc:1428] Setting key 34052821eea8ff606ca01b299e1a0ddf
Sep 28 15:53:58 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610838, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:58.616384 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:58.595734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:50507 (local address 127.22.157.254:32997)
0928 15:53:58.596250 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:58.596257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:58.597323 (+ 1066us) server_negotiation.cc:407] Connection header received
0928 15:53:58.599200 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:58.599208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:58.599303 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:58.599428 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:58.601151 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:58.602357 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:58.604038 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:58.604822 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:58.604974 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:58.609653 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:58.609659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:58.609662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:58.609702 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:58.612790 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:58.613732 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:58.613739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:58.613748 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:58.613892 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:58.614420 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:58.614424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:58.614426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:58.614740 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:58.614914 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:58.615709 (+ 795us) server_negotiation.cc:299] Negotiation successful
0928 15:53:58.615973 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":253,"threads_started":1}
I20260928 15:53:58.617637 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:58.596289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:50507)
0928 15:53:58.597142 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:58.597203 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:58.598916 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:58.599698 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:58.599707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:58.600256 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:58.600962 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:58.600985 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:58.602526 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:58.602530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:58.603877 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:58.603890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:58.605097 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:58.605104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:58.605347 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:58.606225 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:58.606260 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:58.609312 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:58.613021 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:58.613032 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:58.613057 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:58.613564 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:58.614028 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:58.614033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:58.614037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:58.614274 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:58.614886 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:58.614895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:58.615317 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 15:53:58.615612 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 15:53:58.616249 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":374,"threads_started":1}
I20260928 15:53:58.619419 5095 heartbeater.cc:347] Connected to a master server at 127.22.157.254:32997
I20260928 15:53:58.619813 5095 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:58.621007 5095 heartbeater.cc:511] Master 127.22.157.254:32997 requested a full tablet report, sending...
I20260928 15:53:58.623056 4770 ts_manager.cc:194] Registered new tserver with Master: 2a7ba31dcac04400b85f8642f750e569 (127.22.157.194:35389)
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
I20260928 15:53:58.623977 4770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:50507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:53:58.660614 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:32997
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:53:59.006490 5101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:53:59.006794 5101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:53:59.006950 5101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:53:59.014762 5101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:53:59.014896 5101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:53:59.015080 5101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:53:59.025741 5101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:53:59.027119 5101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:53:59.028481 5101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:53:59.042361 5106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:59.042629 5107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:53:59.042330 5109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:53:59.044245 5101 server_base.cc:1034] running on GCE node
I20260928 15:53:59.045084 5101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:53:59.046121 5101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:53:59.047390 5101 hybrid_clock.cc:648] HybridClock initialized: now 1790610839047331 us; error 53 us; skew 500 ppm
I20260928 15:53:59.047816 5101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:53:59 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:53:59.051450 5101 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:53:59.053442 5101 webserver.cc:460] Webserver started at http://127.22.157.195:38483/ using document root <none> and password file <none>
I20260928 15:53:59.054220 5101 fs_manager.cc:359] Metadata directory not provided
I20260928 15:53:59.054369 5101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:53:59.054818 5101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:53:59.058020 5101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "d446ccf3043e4938b1339069583e96e9"
format_stamp: "Formatted at 2026-09-28 15:53:59 on dist-test-slave-1vw0"
server_key: "4aaf8c98d79cddd289bec74ec7fa7e73"
server_key_iv: "bfc15cc9ed81b2a3c9a4aab974d095da"
server_key_version: "encryptionkey@0"
I20260928 15:53:59.058838 5101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "d446ccf3043e4938b1339069583e96e9"
format_stamp: "Formatted at 2026-09-28 15:53:59 on dist-test-slave-1vw0"
server_key: "4aaf8c98d79cddd289bec74ec7fa7e73"
server_key_iv: "bfc15cc9ed81b2a3c9a4aab974d095da"
server_key_version: "encryptionkey@0"
I20260928 15:53:59.061024 4960 heartbeater.cc:503] Master 127.22.157.254:32997 was elected leader, sending a full tablet report...
I20260928 15:53:59.064399 5101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:53:59.068303 5116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:59.070003 5101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:53:59.070217 5101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "d446ccf3043e4938b1339069583e96e9"
format_stamp: "Formatted at 2026-09-28 15:53:59 on dist-test-slave-1vw0"
server_key: "4aaf8c98d79cddd289bec74ec7fa7e73"
server_key_iv: "bfc15cc9ed81b2a3c9a4aab974d095da"
server_key_version: "encryptionkey@0"
I20260928 15:53:59.070464 5101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:53:59.084030 5101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:53:59.089013 5101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:53:59.089326 5101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:53:59.090399 5101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:53:59.091876 5101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:53:59.091987 5101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:59.092151 5101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:53:59.092231 5101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:53:59.131233 5101 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:33835
I20260928 15:53:59.131332 5230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:33835 every 8 connection(s)
I20260928 15:53:59.132644 5101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:53:59.138046 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 5101
I20260928 15:53:59.138280 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:53:59.138741 23159 external_mini_cluster.cc:1428] Setting key 6085a6b2fdb6f7f8a394ed64edd05459
Sep 28 15:53:59 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610839, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:53:59.154604 5233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.135415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:59571 (local address 127.22.157.254:32997)
0928 15:53:59.135898 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.135903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.136788 (+ 885us) server_negotiation.cc:407] Connection header received
0928 15:53:59.138810 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.138815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.138923 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.139045 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:59.141040 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.142100 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.143696 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.144098 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.144195 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.147783 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:59.147788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:59.147792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:59.147827 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:59.150754 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.151568 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.151574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.151578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.151726 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.152181 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.152185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.152188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.152568 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:59.152792 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.153853 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.154108 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
I20260928 15:53:59.155786 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.135865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:59571)
0928 15:53:59.136590 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.136636 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:59.138507 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:59.139330 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:59.139344 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:59.139977 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:53:59.140818 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.140840 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.142263 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.142267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.143549 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.143562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.144290 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.144294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.144500 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.145284 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:53:59.145307 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:53:59.147546 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:53:59.150925 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:59.150933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:59.150948 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:59.151417 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:59.151832 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:53:59.151835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:53:59.151838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:53:59.152043 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:53:59.152810 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:53:59.152817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:53:59.153321 (+ 504us) client_negotiation.cc:769] Sending connection context
0928 15:53:59.153690 (+ 369us) client_negotiation.cc:240] Negotiation successful
0928 15:53:59.154383 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":277,"threads_started":1}
I20260928 15:53:59.157299 5231 heartbeater.cc:347] Connected to a master server at 127.22.157.254:32997
I20260928 15:53:59.157684 5231 heartbeater.cc:464] Registering TS with master...
I20260928 15:53:59.158564 5231 heartbeater.cc:511] Master 127.22.157.254:32997 requested a full tablet report, sending...
I20260928 15:53:59.160173 4770 ts_manager.cc:194] Registered new tserver with Master: d446ccf3043e4938b1339069583e96e9 (127.22.157.195:33835)
I20260928 15:53:59.161209 4770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:59571
I20260928 15:53:59.168634 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:53:59.188148 5233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.174007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.22.157.254:32997)
0928 15:53:59.174300 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.174310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.174732 (+ 422us) server_negotiation.cc:407] Connection header received
0928 15:53:59.174990 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.174996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.175124 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.175218 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:59.176651 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.177479 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.178903 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.179342 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.179449 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.182151 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:59.182156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:59.182159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:59.182194 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:59.185320 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.186131 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.186136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.186139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.186297 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.186687 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.186697 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.186700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.187075 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:59.187281 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.187606 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.187806 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:53:59.192642 4770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:53:59.199848 4770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:53:59.228416 5246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.213729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:33835 (local address 127.0.0.1:34012)
0928 15:53:59.214603 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.214653 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:59.214870 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:59.216443 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:59.216451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:59.216489 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:59.217052 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.217069 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.220108 (+ 3039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.220117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.223900 (+ 3783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.223925 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.226776 (+ 2851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.226786 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.227138 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.227226 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 15:53:59.227392 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 15:53:59.227700 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":40,"thread_start_us":2636,"threads_started":1}
I20260928 15:53:59.228962 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.213764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.22.157.194:35389)
0928 15:53:59.217590 (+ 3826us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.217602 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.217639 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:53:59.217759 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.217770 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.218082 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.218969 (+ 887us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:59.219222 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.221317 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.224619 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.226688 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.227199 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.227414 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.228419 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.228619 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3637,"thread_start_us":3542,"threads_started":1}
I20260928 15:53:59.228794 5245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.213761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:35389 (local address 127.0.0.1:52830)
0928 15:53:59.216815 (+ 3054us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.216841 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:59.217021 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:59.218340 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:59.218345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:59.218371 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:59.218771 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.218783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.221552 (+ 2769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.221560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.224316 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.224337 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.226927 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.226936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.228018 (+ 1082us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.228052 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:53:59.228150 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:53:59.228300 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2825,"mutex_wait_us":878,"thread_start_us":375,"threads_started":1}
I20260928 15:53:59.229835 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.214375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.22.157.195:33835)
0928 15:53:59.215606 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.215619 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.215663 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:53:59.215791 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.215803 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.216138 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.217297 (+ 1159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:59.217621 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.219886 (+ 2265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.224199 (+ 4313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.226438 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.228973 (+ 2535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.229205 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.229283 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.229511 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1013,"thread_start_us":867,"threads_started":1}
I20260928 15:53:59.233806 5164 tablet_service.cc:1467] Processing CreateTablet for tablet 6b191fb0a4c34ca695856b84d1ec00c8 (DEFAULT_TABLE table=test-table [id=5a8118280dc3414e85acbe974f0348d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:53:59.233735 5030 tablet_service.cc:1467] Processing CreateTablet for tablet 6b191fb0a4c34ca695856b84d1ec00c8 (DEFAULT_TABLE table=test-table [id=5a8118280dc3414e85acbe974f0348d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:53:59.235380 5164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b191fb0a4c34ca695856b84d1ec00c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:59.235383 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b191fb0a4c34ca695856b84d1ec00c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:59.237294 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.216825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43444)
0928 15:53:59.228098 (+ 11273us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.228120 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:59.228230 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:59.230554 (+ 2324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:59.230559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:59.230587 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:59.230971 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.230987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.233493 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.233501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.234924 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.234938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.236388 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.236395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.236564 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.236599 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 15:53:59.236724 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 15:53:59.236904 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11141,"thread_start_us":10964,"threads_started":1}
I20260928 15:53:59.237288 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.217511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.22.157.193:35657)
0928 15:53:59.229833 (+ 12322us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.229842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.229875 (+ 33us) server_negotiation.cc:407] Connection header received
0928 15:53:59.230014 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.230025 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.230308 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.230544 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:59.231869 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.233341 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.235093 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.236166 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.236388 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.236583 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.236809 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.236963 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12143,"thread_start_us":11994,"threads_started":1}
I20260928 15:53:59.241915 4895 tablet_service.cc:1467] Processing CreateTablet for tablet 6b191fb0a4c34ca695856b84d1ec00c8 (DEFAULT_TABLE table=test-table [id=5a8118280dc3414e85acbe974f0348d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:53:59.243569 4895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b191fb0a4c34ca695856b84d1ec00c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:53:59.251647 5251 tablet_bootstrap.cc:492] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: Bootstrap starting.
I20260928 15:53:59.253789 5250 tablet_bootstrap.cc:492] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: Bootstrap starting.
I20260928 15:53:59.257153 5251 tablet_bootstrap.cc:654] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:59.258029 5250 tablet_bootstrap.cc:654] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:59.258436 5251 log.cc:824] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:59.259466 5250 log.cc:824] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:59.260031 5253 tablet_bootstrap.cc:492] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Bootstrap starting.
I20260928 15:53:59.262266 5251 tablet_bootstrap.cc:492] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: No bootstrap required, opened a new log
I20260928 15:53:59.262585 5251 ts_tablet_manager.cc:1397] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 15:53:59.263602 5250 tablet_bootstrap.cc:492] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: No bootstrap required, opened a new log
I20260928 15:53:59.263886 5250 ts_tablet_manager.cc:1397] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260928 15:53:59.264173 5253 tablet_bootstrap.cc:654] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Neither blocks nor log segments found. Creating new log.
I20260928 15:53:59.265648 5253 log.cc:824] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:53:59.272163 5253 tablet_bootstrap.cc:492] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: No bootstrap required, opened a new log
I20260928 15:53:59.272565 5253 ts_tablet_manager.cc:1397] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260928 15:53:59.272701 5250 raft_consensus.cc:348] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } }
I20260928 15:53:59.273347 5250 raft_consensus.cc:374] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:59.273516 5250 raft_consensus.cc:729] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d446ccf3043e4938b1339069583e96e9, State: Initialized, Role: FOLLOWER
I20260928 15:53:59.274251 5250 consensus_queue.cc:260] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } }
I20260928 15:53:59.275188 5251 raft_consensus.cc:348] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } }
I20260928 15:53:59.275714 5251 raft_consensus.cc:374] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:59.275759 5231 heartbeater.cc:503] Master 127.22.157.254:32997 was elected leader, sending a full tablet report...
I20260928 15:53:59.275849 5251 raft_consensus.cc:729] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7ba31dcac04400b85f8642f750e569, State: Initialized, Role: FOLLOWER
I20260928 15:53:59.276522 5250 ts_tablet_manager.cc:1428] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260928 15:53:59.276561 5251 consensus_queue.cc:260] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } }
I20260928 15:53:59.277792 5095 heartbeater.cc:503] Master 127.22.157.254:32997 was elected leader, sending a full tablet report...
I20260928 15:53:59.280375 5251 ts_tablet_manager.cc:1428] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260928 15:53:59.284925 5253 raft_consensus.cc:348] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } }
I20260928 15:53:59.285387 5253 raft_consensus.cc:374] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:53:59.285512 5253 raft_consensus.cc:729] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8151cbdb27444d0e94e304ff726a159a, State: Initialized, Role: FOLLOWER
I20260928 15:53:59.286170 5253 consensus_queue.cc:260] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } }
I20260928 15:53:59.288019 5253 ts_tablet_manager.cc:1428] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 15:53:59.306691 5256 raft_consensus.cc:482] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:53:59.307034 5256 raft_consensus.cc:504] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } }
I20260928 15:53:59.308972 5256 leader_election.cc:288] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a7ba31dcac04400b85f8642f750e569 (127.22.157.194:35389), 8151cbdb27444d0e94e304ff726a159a (127.22.157.193:35657)
I20260928 15:53:59.315925 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.309475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:48449 (local address 127.22.157.194:35389)
0928 15:53:59.309696 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.309702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.309725 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:53:59.309952 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.309957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.310045 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.310157 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:59.310635 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.311767 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.313732 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.314914 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.315036 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.315101 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.315521 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.315654 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:53:59.315937 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.309225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:35389 (local address 127.22.157.195:48449)
0928 15:53:59.309561 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.309583 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:59.309726 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:59.310181 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:59.310185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:59.310217 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:59.310452 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.310463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.311908 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.311913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.313549 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.313563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.315082 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.315089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.315288 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.315325 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:53:59.315456 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 15:53:59.315621 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260928 15:53:59.316888 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.309432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.22.157.195:34641)
0928 15:53:59.310207 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.310227 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:53:59.310353 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:53:59.310732 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:53:59.310740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:53:59.310774 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:53:59.311153 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.311166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.312521 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.312525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.314371 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:53:59.314387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.316070 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.316076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:53:59.316262 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.316299 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:53:59.316391 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:53:59.316515 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":439,"threads_started":1}
I20260928 15:53:59.316902 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.309563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:34641 (local address 127.22.157.193:35657)
0928 15:53:59.309822 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.309830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.310320 (+ 490us) server_negotiation.cc:407] Connection header received
0928 15:53:59.310485 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.310492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.310583 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.310672 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:53:59.311344 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.312344 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.314564 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.315689 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.315845 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.315911 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.316453 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.316607 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 15:53:59.317422 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b191fb0a4c34ca695856b84d1ec00c8" candidate_uuid: "d446ccf3043e4938b1339069583e96e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7ba31dcac04400b85f8642f750e569" is_pre_election: true
I20260928 15:53:59.318086 5050 raft_consensus.cc:2454] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d446ccf3043e4938b1339069583e96e9 in term 0.
I20260928 15:53:59.318349 4915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b191fb0a4c34ca695856b84d1ec00c8" candidate_uuid: "d446ccf3043e4938b1339069583e96e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8151cbdb27444d0e94e304ff726a159a" is_pre_election: true
I20260928 15:53:59.318934 5117 leader_election.cc:302] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7ba31dcac04400b85f8642f750e569, d446ccf3043e4938b1339069583e96e9; no voters:
I20260928 15:53:59.318964 4915 raft_consensus.cc:2454] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d446ccf3043e4938b1339069583e96e9 in term 0.
I20260928 15:53:59.319455 5256 raft_consensus.cc:2781] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:53:59.319629 5256 raft_consensus.cc:482] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:53:59.319770 5256 raft_consensus.cc:3037] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:59.322256 5256 raft_consensus.cc:504] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } }
I20260928 15:53:59.322959 5256 leader_election.cc:288] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [CANDIDATE]: Term 1 election: Requested vote from peers 2a7ba31dcac04400b85f8642f750e569 (127.22.157.194:35389), 8151cbdb27444d0e94e304ff726a159a (127.22.157.193:35657)
I20260928 15:53:59.323647 4915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b191fb0a4c34ca695856b84d1ec00c8" candidate_uuid: "d446ccf3043e4938b1339069583e96e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8151cbdb27444d0e94e304ff726a159a"
I20260928 15:53:59.323665 5050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b191fb0a4c34ca695856b84d1ec00c8" candidate_uuid: "d446ccf3043e4938b1339069583e96e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7ba31dcac04400b85f8642f750e569"
I20260928 15:53:59.323940 4915 raft_consensus.cc:3037] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:59.323956 5050 raft_consensus.cc:3037] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:53:59.326498 4915 raft_consensus.cc:2454] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d446ccf3043e4938b1339069583e96e9 in term 1.
I20260928 15:53:59.326727 5050 raft_consensus.cc:2454] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d446ccf3043e4938b1339069583e96e9 in term 1.
I20260928 15:53:59.327132 5119 leader_election.cc:302] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8151cbdb27444d0e94e304ff726a159a, d446ccf3043e4938b1339069583e96e9; no voters:
I20260928 15:53:59.327600 5256 raft_consensus.cc:2781] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:53:59.328071 5256 raft_consensus.cc:686] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 1 LEADER]: Becoming Leader. State: Replica: d446ccf3043e4938b1339069583e96e9, State: Running, Role: LEADER
I20260928 15:53:59.328622 5256 consensus_queue.cc:237] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } }
I20260928 15:53:59.333604 4770 catalog_manager.cc:5495] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d446ccf3043e4938b1339069583e96e9 (127.22.157.195). New cstate: current_term: 1 leader_uuid: "d446ccf3043e4938b1339069583e96e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } health_report { overall_health: UNKNOWN } } }
W20260928 15:53:59.345108 5096 tablet.cc:2369] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:53:59.353857 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.348320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.22.157.195:33835)
0928 15:53:59.348516 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.348521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.348537 (+ 16us) server_negotiation.cc:407] Connection header received
0928 15:53:59.348626 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.348629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.348787 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.348927 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:53:59.349314 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.350073 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.351502 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.351906 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.352006 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.352647 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:53:59.353315 (+ 668us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:53:59.353465 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.353576 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.353653 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:53:59.362493 4915 raft_consensus.cc:1261] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [term 1 FOLLOWER]: Refusing update from remote peer d446ccf3043e4938b1339069583e96e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:53:59.362499 5050 raft_consensus.cc:1261] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [term 1 FOLLOWER]: Refusing update from remote peer d446ccf3043e4938b1339069583e96e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:53:59.363572 5260 consensus_queue.cc:1035] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:53:59.363927 5256 consensus_queue.cc:1035] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:53:59.391263 4770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:53:59.391584 4770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:53:59.395747 4770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Sending DeleteTablet for 3 replicas of tablet 6b191fb0a4c34ca695856b84d1ec00c8
I20260928 15:53:59.397138 5030 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b191fb0a4c34ca695856b84d1ec00c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:53:59 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:52830
I20260928 15:53:59.397235 4895 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b191fb0a4c34ca695856b84d1ec00c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:53:59 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:43444
I20260928 15:53:59.397249 5164 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b191fb0a4c34ca695856b84d1ec00c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:53:59 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:34012
I20260928 15:53:59.397639 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 4831
I20260928 15:53:59.398665 5275 tablet_replica.cc:330] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: stopping tablet replica
I20260928 15:53:59.399211 5275 raft_consensus.cc:2229] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:53:59.399966 5275 raft_consensus.cc:2258] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:53:59.407606 5275 ts_tablet_manager.cc:1905] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:53:59.409452 5274 tablet_replica.cc:330] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: stopping tablet replica
I20260928 15:53:59.410413 5273 tablet_replica.cc:330] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: stopping tablet replica
I20260928 15:53:59.410879 5273 raft_consensus.cc:2229] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 28 15:53:59 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:53:59.411454 5273 raft_consensus.cc:2258] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:53:59.412092 5274 raft_consensus.cc:2229] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:53:59.412741 5274 raft_consensus.cc:2258] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:53:59.415380 5273 ts_tablet_manager.cc:1905] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:53:59.417502 5274 ts_tablet_manager.cc:1905] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:53:59.421581 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.399284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.22.157.193:35657)
0928 15:53:59.399522 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.399531 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.399559 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:53:59.399773 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.399778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.399869 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.399957 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:59.402142 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.403410 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.405565 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.406203 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.406324 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.412583 (+ 6259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:59.412599 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:59.412616 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:59.412670 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:59.416866 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.417879 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.417888 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.417892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.418086 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.418575 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.418580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.418584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.419084 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:59.419451 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.420815 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.421092 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:53:59.422614 4955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:59.422720 5273 ts_tablet_manager.cc:1918] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:53:59.422884 5273 log.cc:1196] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/6b191fb0a4c34ca695856b84d1ec00c8
I20260928 15:53:59.423643 5273 ts_tablet_manager.cc:1939] T 6b191fb0a4c34ca695856b84d1ec00c8 P 2a7ba31dcac04400b85f8642f750e569: Deleting consensus metadata
I20260928 15:53:59.425573 5275 ts_tablet_manager.cc:1918] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:53:59.425861 5275 log.cc:1196] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/6b191fb0a4c34ca695856b84d1ec00c8
I20260928 15:53:59.426343 4754 catalog_manager.cc:4841] TS 2a7ba31dcac04400b85f8642f750e569 (127.22.157.194:35389): tablet 6b191fb0a4c34ca695856b84d1ec00c8 (table test-table [id=5a8118280dc3414e85acbe974f0348d1]) successfully deleted
I20260928 15:53:59.427132 5275 ts_tablet_manager.cc:1939] T 6b191fb0a4c34ca695856b84d1ec00c8 P d446ccf3043e4938b1339069583e96e9: Deleting consensus metadata
I20260928 15:53:59.429733 4756 catalog_manager.cc:4841] TS d446ccf3043e4938b1339069583e96e9 (127.22.157.195:33835): tablet 6b191fb0a4c34ca695856b84d1ec00c8 (table test-table [id=5a8118280dc3414e85acbe974f0348d1]) successfully deleted
I20260928 15:53:59.556177 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 4831
W20260928 15:53:59.578030 4756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.157.193:35657 (error 108)
I20260928 15:53:59.578382 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 4966
I20260928 15:53:59.580435 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.579908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43470)
0928 15:53:59.580152 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.580280 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 15:53:59.580811 4756 catalog_manager.cc:4584] TS 8151cbdb27444d0e94e304ff726a159a (127.22.157.193:35657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b191fb0a4c34ca695856b84d1ec00c8: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Sep 28 15:53:59 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:53:59.601925 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.579570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.22.157.194:35389)
0928 15:53:59.579776 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.579781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.579796 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:53:59.579853 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.579857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.579926 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.580055 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:59.581909 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.582695 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.584634 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.585123 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.585228 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.591856 (+ 6628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:59.591876 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:59.591900 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:59.591966 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:59.596644 (+ 4678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.597856 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.597869 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.597874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.598069 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.598654 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.598661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.598665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.599274 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:59.599607 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.601141 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.601469 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:53:59.602881 5090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:59.634618 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.634103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43480)
0928 15:53:59.634329 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.634440 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 15:53:59.636071 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.635506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43494)
0928 15:53:59.635766 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.635870 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 15:53:59.697129 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.696496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43502)
0928 15:53:59.696807 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.696927 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 15:53:59.698388 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.698019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43518)
0928 15:53:59.698182 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.698269 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 15:53:59.760951 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 4966
I20260928 15:53:59.779207 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.778541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43526)
0928 15:53:59.778846 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.778979 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 15:53:59.780491 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 5101
I20260928 15:53:59.781411 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.780747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43538)
0928 15:53:59.781145 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.781238 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275}
Sep 28 15:53:59 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:53:59.804384 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.781657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.22.157.195:33835)
0928 15:53:59.781829 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 15:53:59.781836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:53:59.781941 (+ 105us) server_negotiation.cc:407] Connection header received
0928 15:53:59.782324 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:53:59.782338 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:53:59.782467 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:53:59.782598 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:53:59.784965 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.786318 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.789441 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:53:59.790269 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:53:59.790443 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:53:59.795934 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:53:59.795958 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:53:59.795988 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:53:59.796059 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:53:59.800663 (+ 4604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.801656 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.801669 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.801673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.801904 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:53:59.802428 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:53:59.802434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:53:59.802437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:53:59.802995 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:53:59.803350 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 15:53:59.803686 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 15:53:59.803947 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:53:59.805305 5226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:53:59.934727 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 5101
I20260928 15:53:59.954664 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.954129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43540)
0928 15:53:59.954378 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.954476 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 15:53:59.956033 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:53:59.955475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43552)
0928 15:53:59.955787 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 15:53:59.955876 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260928 15:53:59.956027 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 4738
I20260928 15:53:59.957186 4800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:00.072283 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 4738
I20260928 15:54:00.096000 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:32997
--webserver_interface=127.22.157.254
--webserver_port=37055
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:54:00.446177 5282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:00.446614 5282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:00.446784 5282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:00.454412 5282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:54:00.454602 5282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:00.454730 5282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:00.454840 5282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:54:00.454921 5282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:54:00.466449 5282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:00.468022 5282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:00.469480 5282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:00.496393 5289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:00.496418 5288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:00.496439 5291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:00.497437 5282 server_base.cc:1034] running on GCE node
I20260928 15:54:00.498440 5282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:00.500231 5282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:00.501475 5282 hybrid_clock.cc:648] HybridClock initialized: now 1790610840501416 us; error 39 us; skew 500 ppm
I20260928 15:54:00.501968 5282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:00 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610840, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:00.512208 5282 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:54:00.514722 5282 webserver.cc:460] Webserver started at http://127.22.157.254:37055/ using document root <none> and password file <none>
I20260928 15:54:00.515617 5282 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:00.515764 5282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:00.523118 5282 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:54:00.526907 5299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:00.528690 5282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 15:54:00.529003 5282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "af760abfe67d4c08be7f8d49227da4b8"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "f7d6c0fe1da714ccb880d0696cbc42a5"
server_key_iv: "69ac1a4fe4b7acf45713fd16f85c2bf2"
server_key_version: "encryptionkey@0"
I20260928 15:54:00.530346 5282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:00.548401 5282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:00.555074 5282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:00.555732 5282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:00.580590 5282 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:32997
I20260928 15:54:00.580618 5350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:32997 every 8 connection(s)
I20260928 15:54:00.582408 5282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:54:00.583616 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 5282
I20260928 15:54:00.584510 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:35657
--local_ip_for_outbound_sockets=127.22.157.193
--tserver_master_addrs=127.22.157.254:32997
--webserver_port=33667
--webserver_interface=127.22.157.193
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:54:00.590472 5351 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:54:00.595211 5351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Bootstrap starting.
I20260928 15:54:00.626902 5351 log.cc:824] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:00.643836 5351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:54:00.644555 5351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Bootstrap complete.
I20260928 15:54:00.660295 5351 raft_consensus.cc:348] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:54:00.662147 5351 raft_consensus.cc:729] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af760abfe67d4c08be7f8d49227da4b8, State: Initialized, Role: FOLLOWER
I20260928 15:54:00.663065 5351 consensus_queue.cc:260] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:54:00.663405 5351 raft_consensus.cc:388] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:54:00.663558 5351 raft_consensus.cc:482] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:54:00.663762 5351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:54:00.669821 5351 raft_consensus.cc:504] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:54:00.670581 5351 leader_election.cc:302] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af760abfe67d4c08be7f8d49227da4b8; no voters:
I20260928 15:54:00.671276 5351 leader_election.cc:288] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:54:00.671473 5355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:54:00.672765 5355 raft_consensus.cc:686] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 2 LEADER]: Becoming Leader. State: Replica: af760abfe67d4c08be7f8d49227da4b8, State: Running, Role: LEADER
I20260928 15:54:00.673383 5355 consensus_queue.cc:237] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:54:00.673782 5351 sys_catalog.cc:564] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:54:00.676259 5357 sys_catalog.cc:455] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af760abfe67d4c08be7f8d49227da4b8. Latest consensus state: current_term: 2 leader_uuid: "af760abfe67d4c08be7f8d49227da4b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } } }
I20260928 15:54:00.676767 5357 sys_catalog.cc:458] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:54:00.677505 5356 sys_catalog.cc:455] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "af760abfe67d4c08be7f8d49227da4b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } } }
I20260928 15:54:00.677929 5356 sys_catalog.cc:458] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:54:00.677821 5361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:54:00.700060 5361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:54:00.701663 5361 catalog_manager.cc:1259] Loaded cluster ID: 84b100c5970d4d31a879665cd606f395
I20260928 15:54:00.701779 5361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:54:00.705082 5361 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 15:54:00.724154 5361 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 15:54:00.742172 5361 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:54:00.744249 5361 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Loaded TSK: 0
I20260928 15:54:00.746089 5361 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:54:01.008518 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:01.008946 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:01.009117 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:01.016716 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:01.016933 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:01.017158 5353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:54:01.027768 5353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:35657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=33667
--tserver_master_addrs=127.22.157.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:01.029255 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:01.030619 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:01.045226 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:01.047350 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:01.049012 5353 server_base.cc:1034] running on GCE node
W20260928 15:54:01.049014 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:01.049788 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:01.050843 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:01.052110 5353 hybrid_clock.cc:648] HybridClock initialized: now 1790610841052023 us; error 54 us; skew 500 ppm
I20260928 15:54:01.052471 5353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:01 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610841, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:01.055960 5353 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:54:01.057732 5353 webserver.cc:460] Webserver started at http://127.22.157.193:33667/ using document root <none> and password file <none>
I20260928 15:54:01.058501 5353 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:01.058674 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:01.064770 5353 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:54:01.068030 5389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:01.069643 5353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:54:01.069795 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "8151cbdb27444d0e94e304ff726a159a"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "2b7fed6f9eab1a40e7fab32e40eee98a"
server_key_iv: "b36e1f7f6dcff3e56626e576cd606553"
server_key_version: "encryptionkey@0"
I20260928 15:54:01.070864 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:01.091571 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:01.096365 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:01.096628 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:01.098203 5353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:01.100570 5396 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 15:54:01.106371 5396 ts_tablet_manager.cc:1852] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 15:54:01.107445 5396 ts_tablet_manager.cc:1905] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:01.115295 5396 ts_tablet_manager.cc:1918] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:54:01.115540 5396 log.cc:1196] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/6b191fb0a4c34ca695856b84d1ec00c8
I20260928 15:54:01.116122 5396 ts_tablet_manager.cc:1939] T 6b191fb0a4c34ca695856b84d1ec00c8 P 8151cbdb27444d0e94e304ff726a159a: Deleting consensus metadata
I20260928 15:54:01.116573 5353 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 15:54:01.116689 5353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260928 15:54:01.116833 5353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:01.116894 5353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:01.153260 5353 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:35657
I20260928 15:54:01.153355 5502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:35657 every 8 connection(s)
I20260928 15:54:01.154711 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:54:01.159607 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 5353
I20260928 15:54:01.160452 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:35389
--local_ip_for_outbound_sockets=127.22.157.194
--tserver_master_addrs=127.22.157.254:32997
--webserver_port=45581
--webserver_interface=127.22.157.194
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:54:01 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610841, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:01.185257 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:01.158220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:41015)
0928 15:54:01.159077 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:01.159141 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:01.162133 (+ 2992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:01.163691 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:01.163705 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:01.164567 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:54:01.165431 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:01.165464 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:01.167134 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:01.167142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:01.169425 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:01.169444 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:01.171059 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:01.171065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:01.171274 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:01.172250 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:54:01.172275 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:54:01.174813 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:54:01.180179 (+ 5366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:01.180188 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:01.180203 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:01.180865 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:01.181517 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:01.181521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:01.181523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:01.181732 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:01.182918 (+ 1186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:54:01.182925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:54:01.183273 (+ 348us) client_negotiation.cc:769] Sending connection context
0928 15:54:01.183528 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 15:54:01.183985 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":307,"threads_started":1}
I20260928 15:54:01.185876 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:01.157902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:41015 (local address 127.22.157.254:32997)
0928 15:54:01.158524 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:01.158544 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:01.159302 (+ 758us) server_negotiation.cc:407] Connection header received
0928 15:54:01.162758 (+ 3456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:01.162773 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:01.163143 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:01.163452 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:01.165688 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:01.166960 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:01.169583 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:01.170783 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:01.170990 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:01.175132 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:01.175145 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:01.175170 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:01.175262 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:01.179934 (+ 4672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:01.181097 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:01.181109 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:01.181119 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:01.181356 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:01.181892 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:01.181900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:01.181906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:01.182741 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:01.182971 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:01.183855 (+ 884us) server_negotiation.cc:299] Negotiation successful
0928 15:54:01.184373 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":249,"threads_started":1}
I20260928 15:54:01.188570 5503 heartbeater.cc:347] Connected to a master server at 127.22.157.254:32997
I20260928 15:54:01.188925 5503 heartbeater.cc:464] Registering TS with master...
I20260928 15:54:01.189643 5503 heartbeater.cc:511] Master 127.22.157.254:32997 requested a full tablet report, sending...
I20260928 15:54:01.191804 5316 ts_manager.cc:194] Registered new tserver with Master: 8151cbdb27444d0e94e304ff726a159a (127.22.157.193:35657)
I20260928 15:54:01.193421 5316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:41015
W20260928 15:54:01.513685 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:01.514042 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:01.514231 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:01.521992 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:01.522173 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:01.522370 5507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:54:01.532917 5507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:35389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=45581
--tserver_master_addrs=127.22.157.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:01.534430 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:01.535905 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:01.551847 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:01.552136 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:01.552202 5516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:01.553017 5507 server_base.cc:1034] running on GCE node
I20260928 15:54:01.553952 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:01.555087 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:01.556367 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790610841556318 us; error 42 us; skew 500 ppm
I20260928 15:54:01.556886 5507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:01 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610841, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:01.560586 5507 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:54:01.562244 5507 webserver.cc:460] Webserver started at http://127.22.157.194:45581/ using document root <none> and password file <none>
I20260928 15:54:01.562958 5507 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:01.563081 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:01.569880 5507 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 15:54:01.573074 5523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:01.574730 5507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:54:01.574963 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "2a7ba31dcac04400b85f8642f750e569"
format_stamp: "Formatted at 2026-09-28 15:53:58 on dist-test-slave-1vw0"
server_key: "1e2f020bc482d54a468a3103b43027f5"
server_key_iv: "df101ceb49efb26f30170c080afe5bf3"
server_key_version: "encryptionkey@0"
I20260928 15:54:01.576184 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:01.626812 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:01.631841 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:01.632102 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:01.633314 5507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:01.635064 5507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:01.635165 5507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:01.635262 5507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:01.635324 5507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:01.672299 5507 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:35389
I20260928 15:54:01.672354 5635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:35389 every 8 connection(s)
I20260928 15:54:01.674069 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:54:01.683280 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 5507
I20260928 15:54:01.684046 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:33835
--local_ip_for_outbound_sockets=127.22.157.195
--tserver_master_addrs=127.22.157.254:32997
--webserver_port=38483
--webserver_interface=127.22.157.195
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:54:01 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610841, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:01.697852 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:01.677223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:51411 (local address 127.22.157.254:32997)
0928 15:54:01.677417 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:01.677422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:01.678645 (+ 1223us) server_negotiation.cc:407] Connection header received
0928 15:54:01.680792 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:01.680799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:01.680901 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:01.680994 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:01.682817 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:01.683983 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:01.685886 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:01.686703 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:01.686859 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:01.690833 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:01.690844 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:01.690852 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:01.690917 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:01.694374 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:01.695253 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:01.695263 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:01.695267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:01.695418 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:01.695892 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:01.695898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:01.695901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:01.696201 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:01.696365 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:01.697205 (+ 840us) server_negotiation.cc:299] Negotiation successful
0928 15:54:01.697470 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:54:01.698956 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:01.677732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:51411)
0928 15:54:01.678461 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:01.678505 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:01.680448 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:01.681275 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:01.681285 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:01.681942 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:54:01.682617 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:01.682639 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:01.684189 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:01.684197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:01.685708 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:01.685721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:01.686948 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:01.686953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:01.687187 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:01.687954 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:54:01.687983 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:54:01.690558 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:54:01.694580 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:01.694591 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:01.694619 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:01.695115 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:01.695548 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:01.695553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:01.695557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:01.695768 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:01.696338 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:54:01.696346 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:54:01.696797 (+ 451us) client_negotiation.cc:769] Sending connection context
0928 15:54:01.697111 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 15:54:01.697726 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":281,"threads_started":1}
I20260928 15:54:01.700592 5636 heartbeater.cc:347] Connected to a master server at 127.22.157.254:32997
I20260928 15:54:01.700981 5636 heartbeater.cc:464] Registering TS with master...
I20260928 15:54:01.701999 5636 heartbeater.cc:511] Master 127.22.157.254:32997 requested a full tablet report, sending...
I20260928 15:54:01.703737 5316 ts_manager.cc:194] Registered new tserver with Master: 2a7ba31dcac04400b85f8642f750e569 (127.22.157.194:35389)
I20260928 15:54:01.704627 5316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:51411
W20260928 15:54:02.038157 5639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:02.038479 5639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:02.038579 5639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:02.046474 5639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:02.046615 5639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:02.046732 5639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:54:02.057130 5639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:33835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=38483
--tserver_master_addrs=127.22.157.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:02.058477 5639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:02.059818 5639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:02.074396 5644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:02.074497 5645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:02.074469 5647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:02.075353 5639 server_base.cc:1034] running on GCE node
I20260928 15:54:02.076133 5639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:02.077327 5639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:02.078611 5639 hybrid_clock.cc:648] HybridClock initialized: now 1790610842078550 us; error 50 us; skew 500 ppm
I20260928 15:54:02.079010 5639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:02 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610842, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:02.082821 5639 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:54:02.084386 5639 webserver.cc:460] Webserver started at http://127.22.157.195:38483/ using document root <none> and password file <none>
I20260928 15:54:02.085198 5639 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:02.085322 5639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:02.091156 5639 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 15:54:02.094672 5654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:02.096452 5639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:54:02.096670 5639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "d446ccf3043e4938b1339069583e96e9"
format_stamp: "Formatted at 2026-09-28 15:53:59 on dist-test-slave-1vw0"
server_key: "4aaf8c98d79cddd289bec74ec7fa7e73"
server_key_iv: "bfc15cc9ed81b2a3c9a4aab974d095da"
server_key_version: "encryptionkey@0"
I20260928 15:54:02.097846 5639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:02.112130 5639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:02.116842 5639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:02.117161 5639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:02.118245 5639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:02.119726 5639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:02.119853 5639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:02.119990 5639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:02.120079 5639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:02.160019 5639 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:33835
I20260928 15:54:02.160076 5767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:33835 every 8 connection(s)
I20260928 15:54:02.161651 5639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:54:02.164602 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 5639
Sep 28 15:54:02 dist-test-slave-1vw0 krb5kdc[4725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610842, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:02.185338 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.165725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49462 (local address 127.22.157.254:32997)
0928 15:54:02.169184 (+ 3459us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.169188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.169710 (+ 522us) server_negotiation.cc:407] Connection header received
0928 15:54:02.169933 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.169939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.170045 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.170241 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:02.171993 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.172933 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.174496 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.175069 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.175242 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.178957 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:02.178963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:02.178967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:02.179002 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:02.182308 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.183063 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.183074 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.183078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.183229 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.183668 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.183674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.183679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.184005 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:02.184300 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.184648 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.184916 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":3337,"thread_start_us":3203,"threads_started":1}
I20260928 15:54:02.186996 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.164521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:55793 (local address 127.22.157.254:32997)
0928 15:54:02.164760 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.164768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.166660 (+ 1892us) server_negotiation.cc:407] Connection header received
0928 15:54:02.170078 (+ 3418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.170083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.170179 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.170276 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:02.172114 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.173067 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.174971 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.175658 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.175827 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.180394 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:02.180399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:02.180402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:02.180433 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:02.183476 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.184309 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.184317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.184321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.184445 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.184988 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.184996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.184999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.185387 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:02.185542 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.186387 (+ 845us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.186643 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":63}
I20260928 15:54:02.188215 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.165332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:55793)
0928 15:54:02.166440 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.166505 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:02.169775 (+ 3270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:02.170569 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:02.170578 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:02.171136 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:54:02.171929 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.171949 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.173237 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.173242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.174810 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.174831 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.175947 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.175957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.176339 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.177092 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:54:02.177127 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:54:02.180147 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:54:02.183657 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:02.183667 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:02.183689 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:02.184178 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:02.184564 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:02.184569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:02.184573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:02.184872 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:02.185523 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:54:02.185533 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:54:02.185937 (+ 404us) client_negotiation.cc:769] Sending connection context
0928 15:54:02.186261 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 15:54:02.186791 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":434,"threads_started":1}
I20260928 15:54:02.189769 5768 heartbeater.cc:347] Connected to a master server at 127.22.157.254:32997
I20260928 15:54:02.190089 5768 heartbeater.cc:464] Registering TS with master...
I20260928 15:54:02.190864 5768 heartbeater.cc:511] Master 127.22.157.254:32997 requested a full tablet report, sending...
I20260928 15:54:02.192332 5316 ts_manager.cc:194] Registered new tserver with Master: d446ccf3043e4938b1339069583e96e9 (127.22.157.195:33835)
I20260928 15:54:02.193274 5316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:55793
I20260928 15:54:02.196686 5503 heartbeater.cc:503] Master 127.22.157.254:32997 was elected leader, sending a full tablet report...
I20260928 15:54:02.200517 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:54:02.207821 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.201463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.22.157.254:32997)
0928 15:54:02.201686 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.201694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.202184 (+ 490us) server_negotiation.cc:407] Connection header received
0928 15:54:02.202452 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.202462 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.202602 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.202760 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:54:02.203229 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.203971 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.205610 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.206053 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.206195 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.206717 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:54:02.207235 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:54:02.207410 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.207461 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.207550 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:54:02.208793 5316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:54:02.216251 5316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:54:02.241396 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.230156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.22.157.194:35389)
0928 15:54:02.232603 (+ 2447us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.232612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.232647 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:54:02.232802 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.232812 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.233160 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.233375 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:02.234462 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.236251 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.239045 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.240088 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.240303 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.240497 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.240871 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.241054 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2303,"thread_start_us":2280,"threads_started":1}
I20260928 15:54:02.241457 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.229479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:35389 (local address 127.0.0.1:52838)
0928 15:54:02.231771 (+ 2292us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.231811 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:02.232002 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:02.233676 (+ 1674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:02.233685 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:02.233716 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:02.234220 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.234244 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.236390 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.236397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.238845 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.238860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.240328 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.240334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.240541 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.240593 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 15:54:02.240776 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 15:54:02.240987 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2008,"mutex_wait_us":1,"thread_start_us":1851,"threads_started":1}
I20260928 15:54:02.242172 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.229682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43558)
0928 15:54:02.233056 (+ 3374us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.233076 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:02.233205 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:02.234456 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:02.234460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:02.234486 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:02.234913 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.234928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.236930 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.236936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.238975 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.238994 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.241345 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.241353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.241542 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.241574 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:54:02.241671 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 15:54:02.241810 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3253,"thread_start_us":3110,"threads_started":1}
I20260928 15:54:02.242666 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.230715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.22.157.193:35657)
0928 15:54:02.232581 (+ 1866us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.232592 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.233338 (+ 746us) server_negotiation.cc:407] Connection header received
0928 15:54:02.233441 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.233450 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.233731 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.233940 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:02.235135 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.236757 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.239176 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.240796 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.241000 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.241177 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.241791 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.241968 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1654,"thread_start_us":1464,"threads_started":1}
I20260928 15:54:02.244670 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.234195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:33835 (local address 127.0.0.1:34044)
0928 15:54:02.235211 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.235226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:02.235365 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:02.235891 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:02.235896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:02.235921 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:02.236355 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.236370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.238425 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.238431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.240656 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.240672 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.243889 (+ 3217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.243895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.244069 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.244101 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:54:02.244202 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:54:02.244347 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":747,"threads_started":1}
I20260928 15:54:02.244786 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.233551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.22.157.195:33835)
0928 15:54:02.234718 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.234729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.235328 (+ 599us) server_negotiation.cc:407] Connection header received
0928 15:54:02.235535 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.235541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.235718 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.235961 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:02.236567 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.238286 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.242202 (+ 3916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.243663 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.243871 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.244049 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.244274 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.244450 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":823,"threads_started":1}
I20260928 15:54:02.244669 5438 tablet_service.cc:1467] Processing CreateTablet for tablet c8882e2669d64c05b066564cd34a42f8 (DEFAULT_TABLE table=test-table [id=a987b8827c0b457e8555584db8e0e892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:54:02.246202 5438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8882e2669d64c05b066564cd34a42f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:02.246239 5571 tablet_service.cc:1467] Processing CreateTablet for tablet c8882e2669d64c05b066564cd34a42f8 (DEFAULT_TABLE table=test-table [id=a987b8827c0b457e8555584db8e0e892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:54:02.247664 5571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8882e2669d64c05b066564cd34a42f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:02.248587 5703 tablet_service.cc:1467] Processing CreateTablet for tablet c8882e2669d64c05b066564cd34a42f8 (DEFAULT_TABLE table=test-table [id=a987b8827c0b457e8555584db8e0e892]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:54:02.250032 5703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8882e2669d64c05b066564cd34a42f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:02.254308 5784 tablet_bootstrap.cc:492] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Bootstrap starting.
I20260928 15:54:02.258466 5784 tablet_bootstrap.cc:654] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:02.259579 5784 log.cc:824] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:02.262480 5784 tablet_bootstrap.cc:492] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: No bootstrap required, opened a new log
I20260928 15:54:02.262778 5784 ts_tablet_manager.cc:1397] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 15:54:02.264717 5786 tablet_bootstrap.cc:492] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: Bootstrap starting.
I20260928 15:54:02.266753 5787 tablet_bootstrap.cc:492] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: Bootstrap starting.
I20260928 15:54:02.268833 5786 tablet_bootstrap.cc:654] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:02.270231 5786 log.cc:824] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:02.270658 5787 tablet_bootstrap.cc:654] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:02.272102 5787 log.cc:824] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:02.273644 5784 raft_consensus.cc:348] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } }
I20260928 15:54:02.274083 5786 tablet_bootstrap.cc:492] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: No bootstrap required, opened a new log
I20260928 15:54:02.274279 5784 raft_consensus.cc:374] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:02.274452 5786 ts_tablet_manager.cc:1397] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 15:54:02.274437 5784 raft_consensus.cc:729] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8151cbdb27444d0e94e304ff726a159a, State: Initialized, Role: FOLLOWER
I20260928 15:54:02.275225 5784 consensus_queue.cc:260] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } }
I20260928 15:54:02.275921 5787 tablet_bootstrap.cc:492] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: No bootstrap required, opened a new log
I20260928 15:54:02.276225 5787 ts_tablet_manager.cc:1397] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260928 15:54:02.277201 5784 ts_tablet_manager.cc:1428] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260928 15:54:02.283639 5786 raft_consensus.cc:348] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } }
I20260928 15:54:02.284081 5786 raft_consensus.cc:374] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:02.284190 5786 raft_consensus.cc:729] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7ba31dcac04400b85f8642f750e569, State: Initialized, Role: FOLLOWER
I20260928 15:54:02.284921 5786 consensus_queue.cc:260] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } }
I20260928 15:54:02.286456 5636 heartbeater.cc:503] Master 127.22.157.254:32997 was elected leader, sending a full tablet report...
I20260928 15:54:02.287137 5786 ts_tablet_manager.cc:1428] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260928 15:54:02.287405 5787 raft_consensus.cc:348] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } }
I20260928 15:54:02.287935 5787 raft_consensus.cc:374] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:02.288057 5787 raft_consensus.cc:729] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d446ccf3043e4938b1339069583e96e9, State: Initialized, Role: FOLLOWER
I20260928 15:54:02.288801 5787 consensus_queue.cc:260] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } }
I20260928 15:54:02.290045 5768 heartbeater.cc:503] Master 127.22.157.254:32997 was elected leader, sending a full tablet report...
I20260928 15:54:02.290723 5787 ts_tablet_manager.cc:1428] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 15:54:02.371671 5790 raft_consensus.cc:482] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:54:02.372035 5790 raft_consensus.cc:504] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } }
I20260928 15:54:02.373929 5790 leader_election.cc:288] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a7ba31dcac04400b85f8642f750e569 (127.22.157.194:35389), d446ccf3043e4938b1339069583e96e9 (127.22.157.195:33835)
I20260928 15:54:02.383908 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.374479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:33935 (local address 127.22.157.194:35389)
0928 15:54:02.374666 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.374672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.375497 (+ 825us) server_negotiation.cc:407] Connection header received
0928 15:54:02.375778 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.375783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.375858 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.375949 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:02.376887 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.378015 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.381334 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.382695 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.382863 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.382940 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.383488 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.383627 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:54:02.383908 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.374424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:39739 (local address 127.22.157.195:33835)
0928 15:54:02.374654 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.374660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.375518 (+ 858us) server_negotiation.cc:407] Connection header received
0928 15:54:02.375726 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.375733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.375816 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.375935 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:02.376887 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.378016 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.381310 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.382696 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.382863 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.382940 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.383487 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.383643 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:54:02.384191 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.374353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:33835 (local address 127.22.157.193:39739)
0928 15:54:02.375410 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.375427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:02.375564 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:02.375975 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:02.375980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:02.376012 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:02.376677 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.376686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.378197 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.378203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.381097 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.381117 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.383013 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.383024 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.383259 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.383334 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 15:54:02.383442 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:54:02.383660 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":286,"thread_start_us":525,"threads_started":1}
I20260928 15:54:02.384449 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.374311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:35389 (local address 127.22.157.193:33935)
0928 15:54:02.375333 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.375356 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:02.375579 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:02.375975 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:02.375980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:02.376031 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:02.376651 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.376672 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.378197 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.378203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.381097 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:02.381117 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.383001 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.383008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:02.383266 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.383334 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 15:54:02.383442 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:54:02.383660 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":311,"spinlock_wait_cycles":81536,"thread_start_us":575,"threads_started":1}
I20260928 15:54:02.385378 5723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8882e2669d64c05b066564cd34a42f8" candidate_uuid: "8151cbdb27444d0e94e304ff726a159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d446ccf3043e4938b1339069583e96e9" is_pre_election: true
I20260928 15:54:02.385550 5591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8882e2669d64c05b066564cd34a42f8" candidate_uuid: "8151cbdb27444d0e94e304ff726a159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7ba31dcac04400b85f8642f750e569" is_pre_election: true
I20260928 15:54:02.385939 5723 raft_consensus.cc:2454] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8151cbdb27444d0e94e304ff726a159a in term 0.
I20260928 15:54:02.386063 5591 raft_consensus.cc:2454] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8151cbdb27444d0e94e304ff726a159a in term 0.
I20260928 15:54:02.386682 5392 leader_election.cc:302] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a7ba31dcac04400b85f8642f750e569, 8151cbdb27444d0e94e304ff726a159a, d446ccf3043e4938b1339069583e96e9; no voters:
I20260928 15:54:02.387140 5790 raft_consensus.cc:2781] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:54:02.387363 5790 raft_consensus.cc:482] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:54:02.387499 5790 raft_consensus.cc:3037] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:02.390201 5790 raft_consensus.cc:504] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } }
I20260928 15:54:02.390815 5790 leader_election.cc:288] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [CANDIDATE]: Term 1 election: Requested vote from peers 2a7ba31dcac04400b85f8642f750e569 (127.22.157.194:35389), d446ccf3043e4938b1339069583e96e9 (127.22.157.195:33835)
I20260928 15:54:02.391350 5591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8882e2669d64c05b066564cd34a42f8" candidate_uuid: "8151cbdb27444d0e94e304ff726a159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a7ba31dcac04400b85f8642f750e569"
I20260928 15:54:02.391356 5723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8882e2669d64c05b066564cd34a42f8" candidate_uuid: "8151cbdb27444d0e94e304ff726a159a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d446ccf3043e4938b1339069583e96e9"
I20260928 15:54:02.391584 5591 raft_consensus.cc:3037] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:02.391650 5723 raft_consensus.cc:3037] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:02.395054 5591 raft_consensus.cc:2454] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8151cbdb27444d0e94e304ff726a159a in term 1.
I20260928 15:54:02.395046 5723 raft_consensus.cc:2454] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8151cbdb27444d0e94e304ff726a159a in term 1.
I20260928 15:54:02.395589 5390 leader_election.cc:302] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a7ba31dcac04400b85f8642f750e569, 8151cbdb27444d0e94e304ff726a159a; no voters:
I20260928 15:54:02.395961 5790 raft_consensus.cc:2781] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:54:02.396420 5790 raft_consensus.cc:686] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 1 LEADER]: Becoming Leader. State: Replica: 8151cbdb27444d0e94e304ff726a159a, State: Running, Role: LEADER
I20260928 15:54:02.396993 5790 consensus_queue.cc:237] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } }
I20260928 15:54:02.402004 5315 catalog_manager.cc:5495] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8151cbdb27444d0e94e304ff726a159a (127.22.157.193). New cstate: current_term: 1 leader_uuid: "8151cbdb27444d0e94e304ff726a159a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8151cbdb27444d0e94e304ff726a159a" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 } health_report { overall_health: UNKNOWN } } }
W20260928 15:54:02.407229 5504 tablet.cc:2369] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:54:02.413303 5769 tablet.cc:2369] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:54:02.426529 5637 tablet.cc:2369] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:54:02.440277 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.433978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.22.157.193:35657)
0928 15:54:02.434215 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.434222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.434336 (+ 114us) server_negotiation.cc:407] Connection header received
0928 15:54:02.434618 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.434622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.434767 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.434948 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:54:02.435360 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.436193 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.437768 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.438220 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.438381 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.439004 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:54:02.439575 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:54:02.439750 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.439866 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.439964 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:54:02.449882 5591 raft_consensus.cc:1261] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [term 1 FOLLOWER]: Refusing update from remote peer 8151cbdb27444d0e94e304ff726a159a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:54:02.449882 5723 raft_consensus.cc:1261] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [term 1 FOLLOWER]: Refusing update from remote peer 8151cbdb27444d0e94e304ff726a159a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:54:02.451078 5790 consensus_queue.cc:1035] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a7ba31dcac04400b85f8642f750e569" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:54:02.451434 5795 consensus_queue.cc:1035] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d446ccf3043e4938b1339069583e96e9" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:54:02.482553 5315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:54:02.482896 5315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:54:02.487042 5315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Sending DeleteTablet for 3 replicas of tablet c8882e2669d64c05b066564cd34a42f8
I20260928 15:54:02.488502 5571 tablet_service.cc:1514] Processing DeleteTablet for tablet c8882e2669d64c05b066564cd34a42f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:54:02 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:52838
I20260928 15:54:02.488890 5703 tablet_service.cc:1514] Processing DeleteTablet for tablet c8882e2669d64c05b066564cd34a42f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:54:02 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:34044
I20260928 15:54:02.489310 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 5353
I20260928 15:54:02.489621 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet c8882e2669d64c05b066564cd34a42f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:54:02 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:43558
I20260928 15:54:02.501122 5810 tablet_replica.cc:330] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: stopping tablet replica
I20260928 15:54:02.502277 5811 tablet_replica.cc:330] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: stopping tablet replica
I20260928 15:54:02.502874 5810 raft_consensus.cc:2229] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:54:02.502986 5811 raft_consensus.cc:2229] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:54:02.503494 5810 raft_consensus.cc:2258] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:54:02.503938 5811 raft_consensus.cc:2258] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:54:02.506795 5810 ts_tablet_manager.cc:1905] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:02.506795 5811 ts_tablet_manager.cc:1905] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:02.507464 5809 tablet_replica.cc:330] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: stopping tablet replica
I20260928 15:54:02.507997 5809 raft_consensus.cc:2229] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:54:02.508605 5809 raft_consensus.cc:2258] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:54:02.514482 5810 ts_tablet_manager.cc:1918] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:54:02.514693 5810 log.cc:1196] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/c8882e2669d64c05b066564cd34a42f8
I20260928 15:54:02.514985 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.491381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.22.157.193:35657)
0928 15:54:02.491646 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.491655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.491684 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:54:02.491765 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.491773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.491892 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.492013 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:02.495066 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.496628 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.499372 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.500349 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.500542 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.504714 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:02.504756 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:02.504777 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:02.504832 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:02.509357 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.510536 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.510546 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.510550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.510775 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.511322 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.511330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.511333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.511849 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:02.513422 (+ 1573us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.513612 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.514586 (+ 974us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 15:54:02.515321 5810 ts_tablet_manager.cc:1939] T c8882e2669d64c05b066564cd34a42f8 P d446ccf3043e4938b1339069583e96e9: Deleting consensus metadata
I20260928 15:54:02.515941 5498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:02.517058 5809 ts_tablet_manager.cc:1905] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:02.517660 5302 catalog_manager.cc:4841] TS d446ccf3043e4938b1339069583e96e9 (127.22.157.195:33835): tablet c8882e2669d64c05b066564cd34a42f8 (table test-table [id=a987b8827c0b457e8555584db8e0e892]) successfully deleted
I20260928 15:54:02.525295 5809 ts_tablet_manager.cc:1918] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:54:02.525507 5809 log.cc:1196] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/c8882e2669d64c05b066564cd34a42f8
I20260928 15:54:02.526111 5809 ts_tablet_manager.cc:1939] T c8882e2669d64c05b066564cd34a42f8 P 2a7ba31dcac04400b85f8642f750e569: Deleting consensus metadata
I20260928 15:54:02.528314 5300 catalog_manager.cc:4841] TS 2a7ba31dcac04400b85f8642f750e569 (127.22.157.194:35389): tablet c8882e2669d64c05b066564cd34a42f8 (table test-table [id=a987b8827c0b457e8555584db8e0e892]) successfully deleted
I20260928 15:54:02.632091 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 5353
W20260928 15:54:02.652314 5302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.157.193:35657 (error 108)
I20260928 15:54:02.652485 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 5507
I20260928 15:54:02.655186 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.654605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43592)
0928 15:54:02.654818 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.654964 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 15:54:02.655519 5302 catalog_manager.cc:4584] TS 8151cbdb27444d0e94e304ff726a159a (127.22.157.193:35657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8882e2669d64c05b066564cd34a42f8: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
I20260928 15:54:02.668974 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.654038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.22.157.194:35389)
0928 15:54:02.654261 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.654266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.654281 (+ 15us) server_negotiation.cc:407] Connection header received
0928 15:54:02.654333 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.654336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.654461 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.654570 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:02.656453 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.657334 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.658783 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.659193 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.659294 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.662457 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:02.662474 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:02.662492 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:02.662537 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:02.665923 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.666770 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.666778 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.666781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.666919 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.667295 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.667299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.667300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.667712 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:02.667944 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.668276 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.668454 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:54:02.669759 5631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:02.709350 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.708830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43594)
0928 15:54:02.709074 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.709190 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 15:54:02.710575 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.710166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43610)
0928 15:54:02.710342 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.710430 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 15:54:02.764257 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 5507
I20260928 15:54:02.771512 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.770858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43626)
0928 15:54:02.771124 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.771255 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 15:54:02.772866 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.772305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43638)
0928 15:54:02.772507 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.772648 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 15:54:02.783735 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 5639
I20260928 15:54:02.798898 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.784576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.22.157.195:33835)
0928 15:54:02.784812 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:02.784819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:02.784918 (+ 99us) server_negotiation.cc:407] Connection header received
0928 15:54:02.785160 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:02.785166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:02.785262 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:02.785373 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:02.786754 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.787537 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.789015 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:02.789443 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:02.789603 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:02.792542 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:02.792559 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:02.792576 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:02.792614 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:02.795672 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.796543 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.796551 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.796554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.796714 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:02.797263 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:02.797267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:02.797269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:02.797703 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:02.797935 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:02.798373 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 15:54:02.798585 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:54:02.799726 5763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:02.854077 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.853410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43646)
0928 15:54:02.853659 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.853771 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 15:54:02.855424 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:02.854963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43652)
0928 15:54:02.855206 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:02.855297 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260928 15:54:02.894130 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 5639
I20260928 15:54:02.912141 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 5282
I20260928 15:54:02.913755 5346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:03.034797 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:03.034077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35657 (local address 127.0.0.1:43668)
0928 15:54:03.034336 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:03.034491 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 15:54:03.035092 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 5282
I20260928 15:54:03.059275 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:32997
--webserver_interface=127.22.157.254
--webserver_port=37055
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:54:03.415936 5817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:03.416316 5817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:03.416492 5817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:03.423964 5817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:54:03.424147 5817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:03.424238 5817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:03.424348 5817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:54:03.424418 5817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:54:03.434942 5817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=37055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:03.436568 5817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:03.438189 5817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:03.451246 5822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:03.451359 5825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:03.451385 5823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:03.452370 5817 server_base.cc:1034] running on GCE node
I20260928 15:54:03.453459 5817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:03.455143 5817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:03.456404 5817 hybrid_clock.cc:648] HybridClock initialized: now 1790610843456298 us; error 128 us; skew 500 ppm
I20260928 15:54:03.456900 5817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:03 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610843, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:03.460368 5817 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:54:03.462379 5817 webserver.cc:460] Webserver started at http://127.22.157.254:37055/ using document root <none> and password file <none>
I20260928 15:54:03.463171 5817 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:03.463331 5817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:03.469703 5817 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:54:03.473275 5832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:03.474862 5817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 15:54:03.475080 5817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "af760abfe67d4c08be7f8d49227da4b8"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "f7d6c0fe1da714ccb880d0696cbc42a5"
server_key_iv: "69ac1a4fe4b7acf45713fd16f85c2bf2"
server_key_version: "encryptionkey@0"
I20260928 15:54:03.476194 5817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:03.498858 5817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:03.503695 5817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:03.503942 5817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:03.524140 5817 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:32997
I20260928 15:54:03.524160 5883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:32997 every 8 connection(s)
I20260928 15:54:03.525903 5817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:54:03.527267 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 5817
I20260928 15:54:03.528137 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:35657
--local_ip_for_outbound_sockets=127.22.157.193
--tserver_master_addrs=127.22.157.254:32997
--webserver_port=33667
--webserver_interface=127.22.157.193
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:54:03.533658 5884 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:54:03.537205 5884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Bootstrap starting.
I20260928 15:54:03.558918 5884 log.cc:824] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:03.577708 5884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:54:03.578318 5884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8: Bootstrap complete.
I20260928 15:54:03.588336 5884 raft_consensus.cc:348] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:54:03.589445 5884 raft_consensus.cc:729] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: af760abfe67d4c08be7f8d49227da4b8, State: Initialized, Role: FOLLOWER
I20260928 15:54:03.590183 5884 consensus_queue.cc:260] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:54:03.590463 5884 raft_consensus.cc:388] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:54:03.590613 5884 raft_consensus.cc:482] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:54:03.590814 5884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:54:03.594967 5884 raft_consensus.cc:504] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:54:03.595541 5884 leader_election.cc:302] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af760abfe67d4c08be7f8d49227da4b8; no voters:
I20260928 15:54:03.596272 5884 leader_election.cc:288] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:54:03.596431 5888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:54:03.597746 5888 raft_consensus.cc:686] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [term 3 LEADER]: Becoming Leader. State: Replica: af760abfe67d4c08be7f8d49227da4b8, State: Running, Role: LEADER
I20260928 15:54:03.598412 5888 consensus_queue.cc:237] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } }
I20260928 15:54:03.598752 5884 sys_catalog.cc:564] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:54:03.601377 5889 sys_catalog.cc:455] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "af760abfe67d4c08be7f8d49227da4b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } } }
I20260928 15:54:03.601755 5889 sys_catalog.cc:458] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:54:03.601070 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af760abfe67d4c08be7f8d49227da4b8. Latest consensus state: current_term: 3 leader_uuid: "af760abfe67d4c08be7f8d49227da4b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af760abfe67d4c08be7f8d49227da4b8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 32997 } } }
I20260928 15:54:03.602479 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P af760abfe67d4c08be7f8d49227da4b8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:54:03.604568 5895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:54:03.614176 5895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:54:03.616197 5895 catalog_manager.cc:1259] Loaded cluster ID: 84b100c5970d4d31a879665cd606f395
I20260928 15:54:03.616426 5895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 15:54:03.618917 5895 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8cb2f799ad google::LogMessage::Fail() at ??:0
@ 0x7f8cb2f7c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8cb2f794de google::LogMessage::Flush() at ??:0
@ 0x7f8cb2f7aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8cc89ac512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8cc89a9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8cc8a31610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8cc8a31439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8cb5bc5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8cb5bc891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8cb5bc8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8cb5ba3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8cb8e7d6db start_thread at ??:0
@ 0x7f8cb1e1361f clone at ??:0
W20260928 15:54:03.970903 5886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:03.971273 5886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:03.971393 5886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:03.979025 5886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:03.979163 5886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:03.979282 5886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:54:03.989889 5886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:35657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=33667
--tserver_master_addrs=127.22.157.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:03.991276 5886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:03.992662 5886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:04.006968 5925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:04.007035 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:04.006997 5928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:04.008980 5886 server_base.cc:1034] running on GCE node
I20260928 15:54:04.009831 5886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:04.011003 5886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:04.012255 5886 hybrid_clock.cc:648] HybridClock initialized: now 1790610844012185 us; error 66 us; skew 500 ppm
I20260928 15:54:04.012619 5886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:04 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610844, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:04.016575 5886 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:54:04.018306 5886 webserver.cc:460] Webserver started at http://127.22.157.193:33667/ using document root <none> and password file <none>
I20260928 15:54:04.019094 5886 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:04.019253 5886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:04.025521 5886 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:54:04.028878 5935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:04.030396 5886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:54:04.030694 5886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "8151cbdb27444d0e94e304ff726a159a"
format_stamp: "Formatted at 2026-09-28 15:53:57 on dist-test-slave-1vw0"
server_key: "2b7fed6f9eab1a40e7fab32e40eee98a"
server_key_iv: "b36e1f7f6dcff3e56626e576cd606553"
server_key_version: "encryptionkey@0"
I20260928 15:54:04.031829 5886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:04.046492 5886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:04.051192 5886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:04.051519 5886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:04.052644 5886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:04.054728 5942 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 15:54:04.060674 5942 ts_tablet_manager.cc:1852] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 15:54:04.061811 5942 ts_tablet_manager.cc:1905] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:04.070798 5942 ts_tablet_manager.cc:1918] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:54:04.071081 5942 log.cc:1196] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/c8882e2669d64c05b066564cd34a42f8
I20260928 15:54:04.071688 5942 ts_tablet_manager.cc:1939] T c8882e2669d64c05b066564cd34a42f8 P 8151cbdb27444d0e94e304ff726a159a: Deleting consensus metadata
I20260928 15:54:04.072170 5886 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 15:54:04.072355 5886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.001s sys 0.000s
I20260928 15:54:04.072535 5886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:04.072633 5886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:04.110020 5886 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:35657
I20260928 15:54:04.110074 6049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:35657 every 8 connection(s)
I20260928 15:54:04.111593 5886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:54:04.112864 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 5886
I20260928 15:54:04.113673 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:35389
--local_ip_for_outbound_sockets=127.22.157.194
--tserver_master_addrs=127.22.157.254:32997
--webserver_port=45581
--webserver_interface=127.22.157.194
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:54:04.118165 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:04.115420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:37417)
0928 15:54:04.116413 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:04.116777 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":348,"threads_started":1}
W20260928 15:54:04.119035 6050 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:32997 (0 consecutive failures): Network error: Failed to ping master at 127.22.157.254:32997: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
I20260928 15:54:04.120237 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:04.119745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:35401)
0928 15:54:04.119958 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:04.120063 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 15:54:04.121426 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:04.120943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:36233)
0928 15:54:04.121140 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:04.121252 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:54:04.121732 6050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:54:04.466033 6053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:04.466365 6053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:04.466534 6053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:04.474432 6053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:04.474560 6053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:04.474746 6053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:54:04.484867 6053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:35389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=45581
--tserver_master_addrs=127.22.157.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:04.486297 6053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:04.487746 6053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:04.502058 6058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:04.502007 6059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:04.502203 6061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:04.503001 6053 server_base.cc:1034] running on GCE node
I20260928 15:54:04.503686 6053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:04.504688 6053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:04.505996 6053 hybrid_clock.cc:648] HybridClock initialized: now 1790610844505911 us; error 59 us; skew 500 ppm
I20260928 15:54:04.506397 6053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:04 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610844, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:04.509850 6053 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:54:04.511515 6053 webserver.cc:460] Webserver started at http://127.22.157.194:45581/ using document root <none> and password file <none>
I20260928 15:54:04.512282 6053 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:04.512446 6053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:04.518493 6053 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:54:04.521503 6068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:04.523010 6053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:54:04.523209 6053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "2a7ba31dcac04400b85f8642f750e569"
format_stamp: "Formatted at 2026-09-28 15:53:58 on dist-test-slave-1vw0"
server_key: "1e2f020bc482d54a468a3103b43027f5"
server_key_iv: "df101ceb49efb26f30170c080afe5bf3"
server_key_version: "encryptionkey@0"
I20260928 15:54:04.524273 6053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:04.549156 6053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:04.554137 6053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:04.554526 6053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:04.555655 6053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:04.557224 6053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:04.557358 6053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:04.557504 6053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:04.557595 6053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:04.595112 6053 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:35389
I20260928 15:54:04.595211 6180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:35389 every 8 connection(s)
I20260928 15:54:04.596676 6053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:54:04.601709 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:04.599737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:38415)
0928 15:54:04.600509 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:04.600783 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":282,"threads_started":1}
W20260928 15:54:04.602367 6181 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:32997 (0 consecutive failures): Network error: Failed to ping master at 127.22.157.254:32997: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
I20260928 15:54:04.603264 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 6053
I20260928 15:54:04.603307 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:04.602939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:53683)
0928 15:54:04.603111 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:04.603190 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:54:04.603899 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:33835
--local_ip_for_outbound_sockets=127.22.157.195
--tserver_master_addrs=127.22.157.254:32997
--webserver_port=38483
--webserver_interface=127.22.157.195
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:54:04.604324 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:04.603892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:36789)
0928 15:54:04.604064 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:04.604168 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 15:54:04.604629 6181 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:54:04.941228 6184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:04.941628 6184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:04.941792 6184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:04.949455 6184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:04.949582 6184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:04.949779 6184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:54:04.960139 6184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:33835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=38483
--tserver_master_addrs=127.22.157.254:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:04.961582 6184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:04.962991 6184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:04.977849 6190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:04.977903 6192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:04.977906 6189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:04.978773 6184 server_base.cc:1034] running on GCE node
I20260928 15:54:04.979480 6184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:04.980499 6184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:04.981714 6184 hybrid_clock.cc:648] HybridClock initialized: now 1790610844981684 us; error 53 us; skew 500 ppm
I20260928 15:54:04.982061 6184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:04 dist-test-slave-1vw0 krb5kdc[4725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610844, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:04.985400 6184 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:54:04.986861 6184 webserver.cc:460] Webserver started at http://127.22.157.195:38483/ using document root <none> and password file <none>
I20260928 15:54:04.987531 6184 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:04.987703 6184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:04.993839 6184 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:54:04.996979 6199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:04.998556 6184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 15:54:04.998793 6184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "d446ccf3043e4938b1339069583e96e9"
format_stamp: "Formatted at 2026-09-28 15:53:59 on dist-test-slave-1vw0"
server_key: "4aaf8c98d79cddd289bec74ec7fa7e73"
server_key_iv: "bfc15cc9ed81b2a3c9a4aab974d095da"
server_key_version: "encryptionkey@0"
I20260928 15:54:04.999971 6184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:05.021010 6184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:05.025679 6184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:05.026012 6184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:05.027091 6184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:05.028602 6184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:05.028762 6184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:05.028916 6184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:05.029021 6184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:05.066011 6184 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:33835
I20260928 15:54:05.066058 6311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:33835 every 8 connection(s)
I20260928 15:54:05.067570 6184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:54:05.071295 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 6184
I20260928 15:54:05.073300 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:05.070803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:57273)
0928 15:54:05.071734 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:05.072006 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":370,"threads_started":1}
W20260928 15:54:05.074080 6312 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:32997 (0 consecutive failures): Network error: Failed to ping master at 127.22.157.254:32997: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
I20260928 15:54:05.075034 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:05.074682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:59709)
0928 15:54:05.074844 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:05.074918 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 15:54:05.075953 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:05.075603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:56887)
0928 15:54:05.075754 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:05.075841 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 15:54:05.076203 6312 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:54:05.123363 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:05.122498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:49495)
0928 15:54:05.123017 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:05.123125 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":256,"threads_started":1}
I20260928 15:54:05.606240 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:05.605314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:47833)
0928 15:54:05.605816 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:05.605953 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":238,"threads_started":1}
I20260928 15:54:06.077903 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:06.076949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:48799)
0928 15:54:06.077494 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:06.077650 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":272,"threads_started":1}
I20260928 15:54:06.125204 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:06.124386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:37013)
0928 15:54:06.124951 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:06.125039 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":346,"threads_started":1}
I20260928 15:54:06.608229 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:06.607360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:45263)
0928 15:54:06.607920 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:06.608013 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":308,"threads_started":1}
I20260928 15:54:07.080289 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:07.079300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:60851)
0928 15:54:07.079912 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:07.080049 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":262,"threads_started":1}
I20260928 15:54:07.127032 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:07.126210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:53625)
0928 15:54:07.126729 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:07.126835 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":237,"threads_started":1}
I20260928 15:54:07.610395 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:07.609405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:40707)
0928 15:54:07.610111 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:07.610208 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":470,"threads_started":1}
I20260928 15:54:08.082873 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:08.081865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:47883)
0928 15:54:08.082538 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:08.082654 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":426,"threads_started":1}
I20260928 15:54:08.129321 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:08.128414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:59309)
0928 15:54:08.129040 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:08.129153 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":343,"threads_started":1}
I20260928 15:54:08.612185 6326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:08.611394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:44167)
0928 15:54:08.611907 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:08.611990 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260928 15:54:09.084699 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:09.083875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:54985)
0928 15:54:09.084369 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:09.084458 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260928 15:54:09.131181 6328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:09.130433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:49617)
0928 15:54:09.130884 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:09.130973 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":231,"threads_started":1}
I20260928 15:54:09.614187 6329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:09.613367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:40565)
0928 15:54:09.613884 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:09.613979 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":255,"threads_started":1}
I20260928 15:54:10.086666 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:10.085883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:37997)
0928 15:54:10.086395 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:10.086481 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":267,"threads_started":1}
I20260928 15:54:10.132998 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:10.132174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:39577)
0928 15:54:10.132661 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:10.132815 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
I20260928 15:54:10.616232 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:10.615444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:52379)
0928 15:54:10.615960 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:10.616060 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":261,"threads_started":1}
I20260928 15:54:11.088370 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:11.087627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:59941)
0928 15:54:11.088103 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:11.088200 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":240,"threads_started":1}
I20260928 15:54:11.135011 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:11.134124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:35473)
0928 15:54:11.134707 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:11.134800 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":221,"threads_started":1}
I20260928 15:54:11.618520 6335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:11.617444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:36497)
0928 15:54:11.618179 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:11.618284 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":274,"threads_started":1}
I20260928 15:54:12.090304 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:12.089407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:46869)
0928 15:54:12.089979 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:12.090079 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":255,"threads_started":1}
I20260928 15:54:12.137233 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:12.136158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:38325)
0928 15:54:12.136924 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:12.137043 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":382,"threads_started":1}
I20260928 15:54:12.620878 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:12.619760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:55751)
0928 15:54:12.620495 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:12.620618 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":376,"threads_started":1}
I20260928 15:54:13.092206 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:13.091382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:59267)
0928 15:54:13.091925 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:13.092018 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":310,"threads_started":1}
I20260928 15:54:13.139009 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:13.138194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:60343)
0928 15:54:13.138699 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:13.138806 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":255,"threads_started":1}
I20260928 15:54:13.622712 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:13.621862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:34843)
0928 15:54:13.622392 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:13.622481 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":247,"threads_started":1}
I20260928 15:54:14.094246 6342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:14.093416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:51947)
0928 15:54:14.093941 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:14.094043 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":242,"threads_started":1}
I20260928 15:54:14.140956 6343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:14.140083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:38083)
0928 15:54:14.140565 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:14.140714 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
I20260928 15:54:14.624656 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:14.623824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:58023)
0928 15:54:14.624369 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:14.624464 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":307,"threads_started":1}
I20260928 15:54:15.096182 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:15.095361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:37399)
0928 15:54:15.095902 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:15.096009 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
I20260928 15:54:15.142796 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:15.142011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:50997)
0928 15:54:15.142503 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:15.142595 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":257,"threads_started":1}
I20260928 15:54:15.626497 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:15.625750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:44851)
0928 15:54:15.626205 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:15.626308 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":235,"threads_started":1}
I20260928 15:54:16.098169 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:16.097383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:58467)
0928 15:54:16.097878 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:16.097976 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260928 15:54:16.144827 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:16.143917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:60025)
0928 15:54:16.144472 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:16.144586 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":266,"threads_started":1}
I20260928 15:54:16.628590 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:16.627639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:43373)
0928 15:54:16.628279 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:16.628410 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":368,"threads_started":1}
I20260928 15:54:17.100019 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:17.099225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:34405)
0928 15:54:17.099755 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:17.099845 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":286,"threads_started":1}
I20260928 15:54:17.146714 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:17.145823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:50693)
0928 15:54:17.146384 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:17.146510 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":312,"threads_started":1}
I20260928 15:54:17.631009 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:17.629893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:38671)
0928 15:54:17.630606 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:17.630738 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":320,"threads_started":1}
I20260928 15:54:18.102110 6354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:18.101258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:35677)
0928 15:54:18.101796 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:18.101916 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":259,"threads_started":1}
I20260928 15:54:18.148620 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:18.147814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:55925)
0928 15:54:18.148340 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:18.148433 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":275,"threads_started":1}
I20260928 15:54:18.632853 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:18.632019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:33281)
0928 15:54:18.632469 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:18.632551 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260928 15:54:19.104182 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:19.103295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:60901)
0928 15:54:19.103870 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:19.103990 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":329,"threads_started":1}
I20260928 15:54:19.150825 6358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:19.149815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:41271)
0928 15:54:19.150469 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:19.150612 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":262,"threads_started":1}
I20260928 15:54:19.635020 6359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:19.634102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:41271)
0928 15:54:19.634690 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:19.634818 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":336,"threads_started":1}
I20260928 15:54:20.085652 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 5886
I20260928 15:54:20.106287 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:20.105434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:46803)
0928 15:54:20.105965 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:20.106084 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":265,"threads_started":1}
I20260928 15:54:20.152892 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:20.151894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.193:58923)
0928 15:54:20.152508 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:20.152658 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":303,"threads_started":1}
I20260928 15:54:20.637509 6363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:20.636347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.194:56345)
0928 15:54:20.637189 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:20.637299 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":313,"threads_started":1}
I20260928 15:54:21.069832 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:20.087163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.22.157.193:35657)
0928 15:54:20.087838 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:20.087858 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:21.049713 (+961855us) server_negotiation.cc:407] Connection header received
0928 15:54:21.051689 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:21.051702 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:21.052223 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:21.052598 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:21.054356 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:21.055851 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:21.057397 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:21.058232 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:21.058399 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:21.061868 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:21.061877 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:21.061896 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:21.061955 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:21.065468 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:21.066417 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:21.066426 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:21.066434 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:21.066677 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:21.067279 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:21.067283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:21.067285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:21.067947 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:21.068219 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:21.068955 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 15:54:21.069495 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":595,"threads_started":1}
I20260928 15:54:21.071419 6045 generic_service.cc:196] Checking for leaks (request via RPC)
W20260928 15:54:21.087847 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 5886: Timed out: CheckLeaks RPC to 127.22.157.193:35657 timed out after 1.000s (SENT)
I20260928 15:54:21.088128 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 5886
I20260928 15:54:21.107931 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 6053
I20260928 15:54:21.108465 6365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:21.107528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:32997 (local address 127.22.157.195:56557)
0928 15:54:21.108136 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:21.108254 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:32997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":284,"threads_started":1}
I20260928 15:54:21.127372 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:21.109121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.22.157.194:35389)
0928 15:54:21.109754 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:21.109768 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:21.109809 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:54:21.111480 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:21.111489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:21.111958 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:21.112316 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:21.113855 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:21.115222 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:21.116604 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:21.117391 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:21.117547 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:21.120304 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:21.120312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:21.120330 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:21.120372 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:21.123570 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:21.124352 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:21.124359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:21.124367 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:21.124551 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:21.125143 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:21.125147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:21.125150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:21.125798 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:21.126067 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:21.126548 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 15:54:21.127032 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":309,"threads_started":1}
I20260928 15:54:21.129042 6176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:21.216302 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 6053
I20260928 15:54:21.235662 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 6184
I20260928 15:54:21.255781 6368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:21.236929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.22.157.195:33835)
0928 15:54:21.237652 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:21.237668 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:21.237716 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:54:21.239497 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:21.239509 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:21.239969 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:21.240309 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:21.241915 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:21.243229 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:21.244659 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:21.245446 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:21.245594 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:21.248700 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:21.248709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:21.248792 (+ 83us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:21.248839 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:21.252034 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:21.252991 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:21.252999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:21.253007 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:21.253206 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:21.253601 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:21.253605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:21.253607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:21.254238 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:21.254519 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:21.254956 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 15:54:21.255458 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":332,"threads_started":1}
I20260928 15:54:21.257407 6307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:21.344795 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 6184
2026-09-28T15:54:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25937 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:54:21 dist-test-slave-1vw0 krb5kdc[6372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:54:21 dist-test-slave-1vw0 krb5kdc[6372](info): set up 2 sockets
Sep 28 15:54:21 dist-test-slave-1vw0 krb5kdc[6372](info): commencing operation
krb5kdc: starting...
W20260928 15:54:22.498520 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:54:22 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:54:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:54:22.784920 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45363
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:45363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:54:23.155171 6385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:23.155503 6385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:23.155598 6385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:23.163257 6385 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:54:23.163384 6385 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:54:23.163422 6385 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:54:23.163486 6385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:54:23.163543 6385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:23.163616 6385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:23.163664 6385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:54:23.163713 6385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:54:23.174249 6385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:45363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:23.175655 6385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:23.177047 6385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:23.190250 6391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:23.190351 6390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:23.190328 6393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:23.191138 6385 server_base.cc:1034] running on GCE node
I20260928 15:54:23.192098 6385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:23.193802 6385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:23.195092 6385 hybrid_clock.cc:648] HybridClock initialized: now 1790610863195028 us; error 41 us; skew 500 ppm
I20260928 15:54:23.195523 6385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:23 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610863, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:23.208288 6385 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:54:23.210175 6385 webserver.cc:460] Webserver started at http://127.22.157.254:42989/ using document root <none> and password file <none>
I20260928 15:54:23.210996 6385 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:23.211114 6385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:23.211405 6385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:54:23.214871 6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "cd396c9cdade480da35a6e027dc4ae32"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "e7de0dbd095fc8ac16ca70c52bed49e6"
server_key_iv: "ce5083d554bd4fb9e2926235da31ee97"
server_key_version: "encryptionkey@0"
I20260928 15:54:23.215727 6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "cd396c9cdade480da35a6e027dc4ae32"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "e7de0dbd095fc8ac16ca70c52bed49e6"
server_key_iv: "ce5083d554bd4fb9e2926235da31ee97"
server_key_version: "encryptionkey@0"
I20260928 15:54:23.221664 6385 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 15:54:23.225739 6400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:23.227283 6385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:54:23.227453 6385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "cd396c9cdade480da35a6e027dc4ae32"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "e7de0dbd095fc8ac16ca70c52bed49e6"
server_key_iv: "ce5083d554bd4fb9e2926235da31ee97"
server_key_version: "encryptionkey@0"
I20260928 15:54:23.227682 6385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:23.247371 6385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:23.252126 6385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:23.252379 6385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:23.272567 6385 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:45363
I20260928 15:54:23.272594 6451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:45363 every 8 connection(s)
I20260928 15:54:23.274214 6385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:54:23.278932 6452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:23.284143 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 6385
I20260928 15:54:23.284384 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:54:23.285324 23159 external_mini_cluster.cc:1428] Setting key cdf427972375e2863ce05aef01c763cc
I20260928 15:54:23.292294 6452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Bootstrap starting.
I20260928 15:54:23.296406 6452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:23.298430 6452 log.cc:824] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:23.302774 6452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: No bootstrap required, opened a new log
Sep 28 15:54:23 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:23.311375 6452 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:54:23.311846 6452 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:23.311981 6452 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd396c9cdade480da35a6e027dc4ae32, State: Initialized, Role: FOLLOWER
I20260928 15:54:23.312815 6452 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:54:23.313129 6452 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:54:23.313282 6452 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:54:23.313491 6452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:23.314424 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:23.287773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.22.157.254:45363)
0928 15:54:23.288911 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:23.288928 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:23.288981 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:54:23.291060 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:23.291087 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:23.291569 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:23.292020 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:23.294077 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:23.295502 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:23.297661 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:23.298821 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:23.299071 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:23.304697 (+ 5626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:23.304710 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:23.304780 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:23.304841 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:23.308804 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:23.309822 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:23.309832 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:23.309844 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:23.310095 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:23.310595 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:23.310600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:23.310603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:23.311272 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:23.311673 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:23.312226 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 15:54:23.312780 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":495,"threads_started":1}
I20260928 15:54:23.316393 6452 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:54:23.317068 6452 leader_election.cc:302] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd396c9cdade480da35a6e027dc4ae32; no voters:
I20260928 15:54:23.317809 6452 leader_election.cc:288] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:54:23.317921 6457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:54:23.318799 6457 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 1 LEADER]: Becoming Leader. State: Replica: cd396c9cdade480da35a6e027dc4ae32, State: Running, Role: LEADER
I20260928 15:54:23.319450 6457 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:54:23.320628 6452 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:54:23.322763 6458 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd396c9cdade480da35a6e027dc4ae32. Latest consensus state: current_term: 1 leader_uuid: "cd396c9cdade480da35a6e027dc4ae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } } }
I20260928 15:54:23.323259 6458 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: This master's current role is: LEADER
I20260928 15:54:23.324743 6459 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd396c9cdade480da35a6e027dc4ae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } } }
I20260928 15:54:23.325222 6459 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: This master's current role is: LEADER
I20260928 15:54:23.330335 6464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:54:23.336815 6464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:54:23.348294 6464 catalog_manager.cc:1347] Generated new cluster ID: c473759299bc4877840826938e0d9682
I20260928 15:54:23.348428 6464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:54:23.360424 6464 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:54:23.361994 6464 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:54:23.371232 6464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Generated new TSK 0
I20260928 15:54:23.372208 6464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:54:23.445459 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45363
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:54:23.813095 6478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:23.813465 6478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:23.813643 6478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:23.821834 6478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:23.822031 6478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:23.822204 6478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:54:23.833377 6478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:23.834930 6478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:23.836628 6478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:23.851568 6486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:23.851862 6483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:23.851835 6484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:23.852540 6478 server_base.cc:1034] running on GCE node
I20260928 15:54:23.853343 6478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:23.854432 6478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:23.855726 6478 hybrid_clock.cc:648] HybridClock initialized: now 1790610863855633 us; error 71 us; skew 500 ppm
I20260928 15:54:23.856150 6478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:23 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610863, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:23.860015 6478 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:54:23.861796 6478 webserver.cc:460] Webserver started at http://127.22.157.193:34215/ using document root <none> and password file <none>
I20260928 15:54:23.862586 6478 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:23.862706 6478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:23.862982 6478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:54:23.866307 6478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "a785046b3220198b52ac497d54b3dfa1"
server_key_iv: "e2194f4274e1d442f6781c954b001fc7"
server_key_version: "encryptionkey@0"
I20260928 15:54:23.867112 6478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "a785046b3220198b52ac497d54b3dfa1"
server_key_iv: "e2194f4274e1d442f6781c954b001fc7"
server_key_version: "encryptionkey@0"
I20260928 15:54:23.873080 6478 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:54:23.877058 6493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:23.878585 6478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:54:23.878744 6478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "a785046b3220198b52ac497d54b3dfa1"
server_key_iv: "e2194f4274e1d442f6781c954b001fc7"
server_key_version: "encryptionkey@0"
I20260928 15:54:23.878939 6478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:23.895744 6478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:23.900550 6478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:23.900884 6478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:23.901914 6478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:23.903354 6478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:23.903432 6478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:23.903519 6478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:23.903566 6478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:23.938670 6478 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:42881
I20260928 15:54:23.938719 6605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:42881 every 8 connection(s)
I20260928 15:54:23.940165 6478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:54:23.947214 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 6478
I20260928 15:54:23.947471 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:54:23.948359 23159 external_mini_cluster.cc:1428] Setting key 8daf2e41180a33a1788663577e99f58b
Sep 28 15:54:23 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610863, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:23.965973 6608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:23.943334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:34411 (local address 127.22.157.254:45363)
0928 15:54:23.943994 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:23.944003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:23.945134 (+ 1131us) server_negotiation.cc:407] Connection header received
0928 15:54:23.947254 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:23.947261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:23.947359 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:23.947494 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:23.949558 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:23.950641 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:23.953303 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:23.954104 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:23.954300 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:23.959110 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:23.959133 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:23.959137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:23.959174 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:23.962220 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:23.963085 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:23.963094 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:23.963101 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:23.963269 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:23.963750 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:23.963758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:23.963764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:23.964175 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:23.964331 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:23.965220 (+ 889us) server_negotiation.cc:299] Negotiation successful
0928 15:54:23.965494 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":313,"threads_started":1}
I20260928 15:54:23.966681 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:23.944074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:34411)
0928 15:54:23.944961 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:23.945000 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:23.946860 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:23.947862 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:23.947871 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:23.948499 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:54:23.949361 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:23.949389 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:23.950799 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:23.950804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:23.953108 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:23.953130 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:23.954425 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:23.954435 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:23.954817 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:23.955602 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:54:23.955638 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:54:23.958822 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:54:23.962393 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:23.962401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:23.962418 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:23.962932 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:23.963378 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:23.963381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:23.963384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:23.963633 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:23.964297 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:54:23.964304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:54:23.964698 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 15:54:23.965047 (+ 349us) client_negotiation.cc:240] Negotiation successful
0928 15:54:23.965475 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":332,"threads_started":1}
I20260928 15:54:23.968374 6606 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45363
I20260928 15:54:23.968739 6606 heartbeater.cc:464] Registering TS with master...
I20260928 15:54:23.969727 6606 heartbeater.cc:511] Master 127.22.157.254:45363 requested a full tablet report, sending...
I20260928 15:54:23.972360 6417 ts_manager.cc:194] Registered new tserver with Master: 0f3bd8aa63f84b2d9e6d0e4e7a9d17df (127.22.157.193:42881)
I20260928 15:54:23.974270 6417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:34411
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:54:24.017354 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45363
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:54:24.369350 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:24.369716 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:24.369886 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:24.377452 6612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:24.377604 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:24.377795 6612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:54:24.387837 6612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:24.389308 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:24.390691 6612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:24.404698 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:24.404753 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:24.405680 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:24.406224 6612 server_base.cc:1034] running on GCE node
I20260928 15:54:24.407102 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:24.408591 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:24.409870 6612 hybrid_clock.cc:648] HybridClock initialized: now 1790610864409791 us; error 72 us; skew 500 ppm
I20260928 15:54:24.410275 6612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:24 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610864, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:24.413848 6612 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:54:24.415472 6612 webserver.cc:460] Webserver started at http://127.22.157.194:33003/ using document root <none> and password file <none>
I20260928 15:54:24.416255 6612 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:24.416401 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:24.416831 6612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:54:24.420184 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "15dedc1a851244309b98974faeb361b2"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "0f2ea0cade9af2af5daf1b3a0e852239"
server_key_iv: "f1ea2a7f00a62833212f51ffe4f921ff"
server_key_version: "encryptionkey@0"
I20260928 15:54:24.421144 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "15dedc1a851244309b98974faeb361b2"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "0f2ea0cade9af2af5daf1b3a0e852239"
server_key_iv: "f1ea2a7f00a62833212f51ffe4f921ff"
server_key_version: "encryptionkey@0"
I20260928 15:54:24.427316 6612 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 15:54:24.431502 6627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:24.433355 6612 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 15:54:24.433573 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "15dedc1a851244309b98974faeb361b2"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "0f2ea0cade9af2af5daf1b3a0e852239"
server_key_iv: "f1ea2a7f00a62833212f51ffe4f921ff"
server_key_version: "encryptionkey@0"
I20260928 15:54:24.433817 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:24.457103 6612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:24.462581 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:24.462936 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:24.464139 6612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:24.466145 6612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:24.466280 6612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:24.466444 6612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:24.466522 6612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:24.502278 6612 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:37431
I20260928 15:54:24.502324 6739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:37431 every 8 connection(s)
I20260928 15:54:24.503695 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:54:24.505543 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 6612
I20260928 15:54:24.505735 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:54:24.506345 23159 external_mini_cluster.cc:1428] Setting key 25048ae0f4b0d8857785311024af0813
Sep 28 15:54:24 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610864, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:24.526278 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:24.506499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:50125 (local address 127.22.157.254:45363)
0928 15:54:24.507007 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:24.507014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:24.508368 (+ 1354us) server_negotiation.cc:407] Connection header received
0928 15:54:24.510465 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:24.510475 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:24.510577 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:24.510701 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:24.512435 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:24.513633 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:24.515279 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:24.516031 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:24.516205 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:24.520171 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:24.520176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:24.520179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:24.520215 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:24.522995 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:24.523799 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:24.523809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:24.523813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:24.523944 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:24.524387 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:24.524393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:24.524396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:24.524688 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:24.524883 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:24.525628 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 15:54:24.525874 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":265,"threads_started":1}
I20260928 15:54:24.527365 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:24.507225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:50125)
0928 15:54:24.508201 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:24.508253 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:24.510130 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:24.510946 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:24.510955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:24.511586 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:54:24.512233 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:24.512253 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:24.513783 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:24.513786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:24.515125 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:24.515138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:24.516259 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:24.516264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:24.516477 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:24.517272 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:54:24.517300 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:54:24.519892 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:54:24.523157 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:24.523167 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:24.523187 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:24.523661 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:24.524059 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:24.524065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:24.524068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:24.524275 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:24.524851 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:54:24.524859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:54:24.525238 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 15:54:24.525541 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 15:54:24.526139 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":404,"threads_started":1}
I20260928 15:54:24.529036 6740 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45363
I20260928 15:54:24.529408 6740 heartbeater.cc:464] Registering TS with master...
I20260928 15:54:24.530232 6740 heartbeater.cc:511] Master 127.22.157.254:45363 requested a full tablet report, sending...
I20260928 15:54:24.532112 6417 ts_manager.cc:194] Registered new tserver with Master: 15dedc1a851244309b98974faeb361b2 (127.22.157.194:37431)
I20260928 15:54:24.533473 6417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:50125
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:54:24.570132 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45363
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:54:24.943379 6746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:24.943732 6746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:24.943907 6746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:24.951896 6746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:24.952091 6746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:24.952272 6746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:54:24.963136 6746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:24.964601 6746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:24.966089 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:54:24.978782 6606 heartbeater.cc:503] Master 127.22.157.254:45363 was elected leader, sending a full tablet report...
W20260928 15:54:24.984797 6751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:24.984800 6752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:24.985728 6754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:24.985721 6746 server_base.cc:1034] running on GCE node
I20260928 15:54:24.986755 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:24.988281 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:24.989560 6746 hybrid_clock.cc:648] HybridClock initialized: now 1790610864989444 us; error 107 us; skew 500 ppm
I20260928 15:54:24.990056 6746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:24 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610864, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:24.994421 6746 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:54:24.996122 6746 webserver.cc:460] Webserver started at http://127.22.157.195:36021/ using document root <none> and password file <none>
I20260928 15:54:24.996982 6746 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:24.997146 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:24.997460 6746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:54:25.000540 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "be1a624e8a266536c6afe0f4cb893f20"
server_key_iv: "4d8630bcf9c86c5bb014368dd3401b18"
server_key_version: "encryptionkey@0"
I20260928 15:54:25.001394 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "be1a624e8a266536c6afe0f4cb893f20"
server_key_iv: "4d8630bcf9c86c5bb014368dd3401b18"
server_key_version: "encryptionkey@0"
I20260928 15:54:25.007231 6746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:54:25.011309 6761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:25.013098 6746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:54:25.013274 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "be1a624e8a266536c6afe0f4cb893f20"
server_key_iv: "4d8630bcf9c86c5bb014368dd3401b18"
server_key_version: "encryptionkey@0"
I20260928 15:54:25.013501 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:25.032616 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:25.037283 6746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:25.037652 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:25.039093 6746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:25.040642 6746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:25.040762 6746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:25.040898 6746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:25.040967 6746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:25.077750 6746 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:41605
I20260928 15:54:25.077769 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:41605 every 8 connection(s)
I20260928 15:54:25.079190 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:54:25.089203 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 6746
I20260928 15:54:25.089396 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:54:25.089946 23159 external_mini_cluster.cc:1428] Setting key 94304864a00c4f1cec85cadee1a3150a
Sep 28 15:54:25 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610864, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:25.105607 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.082575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:35405 (local address 127.22.157.254:45363)
0928 15:54:25.083142 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.083149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.084256 (+ 1107us) server_negotiation.cc:407] Connection header received
0928 15:54:25.086247 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.086256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.086372 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.086533 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:25.088214 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.089736 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.092025 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.092910 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.093058 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.097340 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:25.097350 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:25.097358 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:25.097414 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:25.101768 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.102707 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.102716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.102722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.102898 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.103341 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.103348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.103353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.103798 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:25.103966 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.104951 (+ 985us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.105201 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":261,"threads_started":1}
I20260928 15:54:25.106968 6877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.083146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:35405)
0928 15:54:25.084074 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:25.084126 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:25.085934 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:25.086728 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:25.086737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:25.087364 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:54:25.087947 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.088008 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.089945 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.089951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.091837 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.091854 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.093504 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.093512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.093734 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.094595 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:54:25.094625 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:54:25.097049 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:54:25.101986 (+ 4937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:25.101999 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:25.102014 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:25.102580 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:25.103030 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:25.103033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:25.103035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:25.103228 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:25.103977 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:54:25.103986 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:54:25.104458 (+ 472us) client_negotiation.cc:769] Sending connection context
0928 15:54:25.104877 (+ 419us) client_negotiation.cc:240] Negotiation successful
0928 15:54:25.105453 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":392,"threads_started":1}
I20260928 15:54:25.108489 6874 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45363
I20260928 15:54:25.108901 6874 heartbeater.cc:464] Registering TS with master...
I20260928 15:54:25.109906 6874 heartbeater.cc:511] Master 127.22.157.254:45363 requested a full tablet report, sending...
I20260928 15:54:25.111657 6417 ts_manager.cc:194] Registered new tserver with Master: ce9cf74d6e734b2ab8dfe1044c21e1d3 (127.22.157.195:41605)
I20260928 15:54:25.112496 6417 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:35405
I20260928 15:54:25.119458 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:54:25.144131 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.126659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.22.157.254:45363)
0928 15:54:25.126900 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.126909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.127342 (+ 433us) server_negotiation.cc:407] Connection header received
0928 15:54:25.127548 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.127556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.127682 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.127811 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:25.129399 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.130814 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.132632 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.133517 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.133685 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.136225 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:25.136235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:25.136243 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:25.136296 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:25.140561 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.141464 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.141473 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.141480 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.141665 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.142098 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.142105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.142119 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.142644 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:25.142971 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.143326 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.143646 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:54:25.149610 6417 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:54:25.157147 6417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:54:25.181298 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.171400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.22.157.195:41605)
0928 15:54:25.172682 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.172690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.172760 (+ 70us) server_negotiation.cc:407] Connection header received
0928 15:54:25.173258 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.173266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.173510 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.173704 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:25.174698 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.176340 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.178155 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.179753 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.179957 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.180148 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.180744 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.180932 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":1532,"threads_started":1}
I20260928 15:54:25.181376 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.170666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.22.157.194:37431)
0928 15:54:25.171381 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.171392 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.172012 (+ 620us) server_negotiation.cc:407] Connection header received
0928 15:54:25.172331 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.172341 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.172629 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.172850 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:25.173562 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.175174 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.178231 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.179753 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.179953 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.180148 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.180810 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.180980 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":379,"threads_started":1}
I20260928 15:54:25.181525 6888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.170673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:37431 (local address 127.0.0.1:47264)
0928 15:54:25.171841 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:25.171914 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:25.172110 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:25.172895 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:25.172905 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:25.172947 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:25.173375 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.173388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.175334 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.175340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.178009 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.178026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.180161 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.180170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.180401 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.180509 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 15:54:25.180645 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:54:25.180916 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":127,"thread_start_us":872,"threads_started":1}
I20260928 15:54:25.182297 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.170765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:41605 (local address 127.0.0.1:46320)
0928 15:54:25.171841 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:25.171914 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:25.172110 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:25.173972 (+ 1862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:25.173977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:25.174000 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:25.174451 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.174463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.176473 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.176478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.177989 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.178002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.180295 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.180300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.180491 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.180515 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 15:54:25.180651 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:54:25.181868 (+ 1217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":39,"thread_start_us":1009,"threads_started":1}
I20260928 15:54:25.183041 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.172369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:35636)
0928 15:54:25.174279 (+ 1910us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:25.174301 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:25.174475 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:25.175693 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:25.175697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:25.175724 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:25.176331 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.176345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.178219 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.178224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.179916 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.179929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.182075 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.182082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.182271 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.182313 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:54:25.182411 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 15:54:25.182559 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1759,"mutex_wait_us":64,"thread_start_us":1607,"threads_started":1}
I20260928 15:54:25.183570 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.172982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.22.157.193:42881)
0928 15:54:25.173786 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.173797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.174619 (+ 822us) server_negotiation.cc:407] Connection header received
0928 15:54:25.174730 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.174739 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.174958 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.175397 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:25.176539 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.178099 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.180111 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.181652 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.181882 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.182076 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.182722 (+ 646us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.182893 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":394,"threads_started":1}
I20260928 15:54:25.185880 6675 tablet_service.cc:1467] Processing CreateTablet for tablet e9e2407ebf82420881eb5e96cb2aaf49 (DEFAULT_TABLE table=test-table [id=4e2da24033a04066a166e5305a2cac13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:54:25.186235 6809 tablet_service.cc:1467] Processing CreateTablet for tablet e9e2407ebf82420881eb5e96cb2aaf49 (DEFAULT_TABLE table=test-table [id=4e2da24033a04066a166e5305a2cac13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:54:25.187214 6675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e2407ebf82420881eb5e96cb2aaf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:25.187800 6809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e2407ebf82420881eb5e96cb2aaf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:25.187507 6541 tablet_service.cc:1467] Processing CreateTablet for tablet e9e2407ebf82420881eb5e96cb2aaf49 (DEFAULT_TABLE table=test-table [id=4e2da24033a04066a166e5305a2cac13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:54:25.188954 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e2407ebf82420881eb5e96cb2aaf49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:25.199344 6893 tablet_bootstrap.cc:492] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: Bootstrap starting.
I20260928 15:54:25.203305 6893 tablet_bootstrap.cc:654] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:25.204747 6894 tablet_bootstrap.cc:492] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Bootstrap starting.
I20260928 15:54:25.205005 6893 log.cc:824] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:25.206792 6895 tablet_bootstrap.cc:492] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Bootstrap starting.
I20260928 15:54:25.209492 6894 tablet_bootstrap.cc:654] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:25.210057 6895 tablet_bootstrap.cc:654] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:25.210943 6894 log.cc:824] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:25.211490 6895 log.cc:824] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:25.213465 6893 tablet_bootstrap.cc:492] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: No bootstrap required, opened a new log
I20260928 15:54:25.213773 6893 ts_tablet_manager.cc:1397] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260928 15:54:25.214680 6894 tablet_bootstrap.cc:492] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: No bootstrap required, opened a new log
I20260928 15:54:25.214948 6894 ts_tablet_manager.cc:1397] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 15:54:25.215663 6895 tablet_bootstrap.cc:492] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: No bootstrap required, opened a new log
I20260928 15:54:25.215939 6895 ts_tablet_manager.cc:1397] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260928 15:54:25.222954 6894 raft_consensus.cc:348] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:25.223677 6894 raft_consensus.cc:374] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:25.223855 6894 raft_consensus.cc:729] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9cf74d6e734b2ab8dfe1044c21e1d3, State: Initialized, Role: FOLLOWER
I20260928 15:54:25.224586 6894 consensus_queue.cc:260] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:25.225311 6893 raft_consensus.cc:348] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:25.225792 6893 raft_consensus.cc:374] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:25.225791 6874 heartbeater.cc:503] Master 127.22.157.254:45363 was elected leader, sending a full tablet report...
I20260928 15:54:25.225903 6893 raft_consensus.cc:729] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15dedc1a851244309b98974faeb361b2, State: Initialized, Role: FOLLOWER
I20260928 15:54:25.226599 6893 consensus_queue.cc:260] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:25.227308 6895 raft_consensus.cc:348] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:25.227893 6895 raft_consensus.cc:374] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:25.228001 6895 raft_consensus.cc:729] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3bd8aa63f84b2d9e6d0e4e7a9d17df, State: Initialized, Role: FOLLOWER
I20260928 15:54:25.228160 6894 ts_tablet_manager.cc:1428] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 15:54:25.228870 6895 consensus_queue.cc:260] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:25.229154 6740 heartbeater.cc:503] Master 127.22.157.254:45363 was elected leader, sending a full tablet report...
I20260928 15:54:25.230094 6893 ts_tablet_manager.cc:1428] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260928 15:54:25.231959 6895 ts_tablet_manager.cc:1428] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Time spent starting tablet: real 0.016s user 0.015s sys 0.002s
I20260928 15:54:25.243047 6901 raft_consensus.cc:482] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:54:25.243386 6901 raft_consensus.cc:504] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:25.245445 6901 leader_election.cc:288] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15dedc1a851244309b98974faeb361b2 (127.22.157.194:37431), ce9cf74d6e734b2ab8dfe1044c21e1d3 (127.22.157.195:41605)
I20260928 15:54:25.252880 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.245849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:45487 (local address 127.22.157.194:37431)
0928 15:54:25.246058 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.246066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.246859 (+ 793us) server_negotiation.cc:407] Connection header received
0928 15:54:25.247104 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.247112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.247197 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.247292 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:25.247887 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.248699 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.250634 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.251621 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.251888 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.251978 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.252353 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.252523 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:54:25.253254 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.245971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:50999 (local address 127.22.157.195:41605)
0928 15:54:25.246198 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.246206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.246959 (+ 753us) server_negotiation.cc:407] Connection header received
0928 15:54:25.247155 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.247158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.247215 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.247315 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:25.248031 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.249049 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.251121 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.252119 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.252276 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.252347 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.252856 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.252993 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:54:25.253265 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.245983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:41605 (local address 127.22.157.193:50999)
0928 15:54:25.246862 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:25.246877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:25.247025 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:25.247352 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:25.247357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:25.247382 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:25.247842 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.247857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.249206 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.249210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.250964 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.250982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.252385 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.252393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.252576 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.252612 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:54:25.252742 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 15:54:25.252891 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":520,"threads_started":1}
I20260928 15:54:25.253762 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.245907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:37431 (local address 127.22.157.193:45487)
0928 15:54:25.246758 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:25.246773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:25.246928 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:25.247315 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:25.247319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:25.247344 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:25.247737 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.247747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.248871 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.248877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.250473 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:25.250486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.251921 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.251928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:25.252099 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.252125 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 15:54:25.253099 (+ 974us) client_negotiation.cc:240] Negotiation successful
0928 15:54:25.253234 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":4,"thread_start_us":492,"threads_started":1}
I20260928 15:54:25.254544 6829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2407ebf82420881eb5e96cb2aaf49" candidate_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" is_pre_election: true
I20260928 15:54:25.254832 6695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2407ebf82420881eb5e96cb2aaf49" candidate_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15dedc1a851244309b98974faeb361b2" is_pre_election: true
I20260928 15:54:25.255234 6829 raft_consensus.cc:2454] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3bd8aa63f84b2d9e6d0e4e7a9d17df in term 0.
I20260928 15:54:25.255406 6695 raft_consensus.cc:2454] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f3bd8aa63f84b2d9e6d0e4e7a9d17df in term 0.
W20260928 15:54:25.255874 6741 tablet.cc:2369] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:54:25.256142 6497 leader_election.cc:302] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f3bd8aa63f84b2d9e6d0e4e7a9d17df, 15dedc1a851244309b98974faeb361b2, ce9cf74d6e734b2ab8dfe1044c21e1d3; no voters:
I20260928 15:54:25.256644 6901 raft_consensus.cc:2781] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:54:25.256848 6901 raft_consensus.cc:482] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:54:25.256986 6901 raft_consensus.cc:3037] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:25.259500 6901 raft_consensus.cc:504] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:25.260203 6901 leader_election.cc:288] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [CANDIDATE]: Term 1 election: Requested vote from peers 15dedc1a851244309b98974faeb361b2 (127.22.157.194:37431), ce9cf74d6e734b2ab8dfe1044c21e1d3 (127.22.157.195:41605)
I20260928 15:54:25.260672 6695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2407ebf82420881eb5e96cb2aaf49" candidate_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15dedc1a851244309b98974faeb361b2"
I20260928 15:54:25.260841 6829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e2407ebf82420881eb5e96cb2aaf49" candidate_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3"
I20260928 15:54:25.260964 6695 raft_consensus.cc:3037] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:25.261101 6829 raft_consensus.cc:3037] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:25.263356 6829 raft_consensus.cc:2454] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3bd8aa63f84b2d9e6d0e4e7a9d17df in term 1.
I20260928 15:54:25.263393 6695 raft_consensus.cc:2454] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f3bd8aa63f84b2d9e6d0e4e7a9d17df in term 1.
I20260928 15:54:25.263798 6497 leader_election.cc:302] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3bd8aa63f84b2d9e6d0e4e7a9d17df, ce9cf74d6e734b2ab8dfe1044c21e1d3; no voters:
I20260928 15:54:25.264115 6901 raft_consensus.cc:2781] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:54:25.264523 6901 raft_consensus.cc:686] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 1 LEADER]: Becoming Leader. State: Replica: 0f3bd8aa63f84b2d9e6d0e4e7a9d17df, State: Running, Role: LEADER
I20260928 15:54:25.265097 6901 consensus_queue.cc:237] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:25.269709 6415 catalog_manager.cc:5495] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f3bd8aa63f84b2d9e6d0e4e7a9d17df (127.22.157.193). New cstate: current_term: 1 leader_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } health_report { overall_health: HEALTHY } } }
I20260928 15:54:25.311486 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.304884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.22.157.193:42881)
0928 15:54:25.305143 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.305151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.305266 (+ 115us) server_negotiation.cc:407] Connection header received
0928 15:54:25.305643 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.305648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.305738 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.305937 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:54:25.306362 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.307172 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.309012 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.309476 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.309635 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.310211 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:54:25.310785 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:54:25.310979 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.311070 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.311168 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 15:54:25.320531 6695 raft_consensus.cc:1261] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [term 1 FOLLOWER]: Refusing update from remote peer 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:54:25.320533 6829 raft_consensus.cc:1261] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 1 FOLLOWER]: Refusing update from remote peer 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:54:25.321655 6904 consensus_queue.cc:1035] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:54:25.322069 6901 consensus_queue.cc:1035] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [LEADER]: Connected to new peer: Peer: permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:54:25.330873 6875 tablet.cc:2369] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:54:25.348122 6415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:54:25.348467 6415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:54:25.352378 6415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Sending DeleteTablet for 3 replicas of tablet e9e2407ebf82420881eb5e96cb2aaf49
I20260928 15:54:25.353683 6809 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e2407ebf82420881eb5e96cb2aaf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:54:25 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:46320
I20260928 15:54:25.353814 6675 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e2407ebf82420881eb5e96cb2aaf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:54:25 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:47264
I20260928 15:54:25.354354 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 6478
I20260928 15:54:25.354978 6541 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e2407ebf82420881eb5e96cb2aaf49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:54:25 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:35636
I20260928 15:54:25.355434 6917 tablet_replica.cc:330] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: stopping tablet replica
I20260928 15:54:25.355906 6917 raft_consensus.cc:2229] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:54:25.356518 6917 raft_consensus.cc:2258] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:54:25.357390 6916 tablet_replica.cc:330] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: stopping tablet replica
I20260928 15:54:25.357918 6916 raft_consensus.cc:2229] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:54:25.358530 6916 raft_consensus.cc:2258] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:54:25.359819 6918 tablet_replica.cc:330] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: stopping tablet replica
I20260928 15:54:25.360247 6918 raft_consensus.cc:2229] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:54:25.360873 6917 ts_tablet_manager.cc:1905] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:25.360989 6918 raft_consensus.cc:2258] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:54:25.363279 6918 ts_tablet_manager.cc:1905] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:25.364825 6916 ts_tablet_manager.cc:1905] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:25.370132 6917 ts_tablet_manager.cc:1918] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 28 15:54:25 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:54:25.371482 6918 ts_tablet_manager.cc:1918] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:54:25.371706 6918 log.cc:1196] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/e9e2407ebf82420881eb5e96cb2aaf49
I20260928 15:54:25.372288 6918 ts_tablet_manager.cc:1939] T e9e2407ebf82420881eb5e96cb2aaf49 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Deleting consensus metadata
I20260928 15:54:25.374455 6402 catalog_manager.cc:4841] TS 0f3bd8aa63f84b2d9e6d0e4e7a9d17df (127.22.157.193:42881): tablet e9e2407ebf82420881eb5e96cb2aaf49 (table test-table [id=4e2da24033a04066a166e5305a2cac13]) successfully deleted
I20260928 15:54:25.374670 6917 log.cc:1196] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/e9e2407ebf82420881eb5e96cb2aaf49
I20260928 15:54:25.375288 6917 ts_tablet_manager.cc:1939] T e9e2407ebf82420881eb5e96cb2aaf49 P 15dedc1a851244309b98974faeb361b2: Deleting consensus metadata
I20260928 15:54:25.377395 6402 catalog_manager.cc:4841] TS 15dedc1a851244309b98974faeb361b2 (127.22.157.194:37431): tablet e9e2407ebf82420881eb5e96cb2aaf49 (table test-table [id=4e2da24033a04066a166e5305a2cac13]) successfully deleted
I20260928 15:54:25.379896 6916 ts_tablet_manager.cc:1918] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:54:25.380163 6916 log.cc:1196] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/e9e2407ebf82420881eb5e96cb2aaf49
I20260928 15:54:25.380542 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.355672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.22.157.193:42881)
0928 15:54:25.355842 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.355850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.359436 (+ 3586us) server_negotiation.cc:407] Connection header received
0928 15:54:25.359769 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.359776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.359869 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.359949 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:25.362155 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.363332 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.365534 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.366314 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.366419 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.372807 (+ 6388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:25.372822 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:25.372842 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:25.372896 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:25.377311 (+ 4415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.378227 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.378235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.378237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.378386 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.378807 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.378811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.378814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.379179 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:25.379517 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.379824 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.380083 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 15:54:25.380816 6916 ts_tablet_manager.cc:1939] T e9e2407ebf82420881eb5e96cb2aaf49 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Deleting consensus metadata
I20260928 15:54:25.381361 6601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:25.382761 6404 catalog_manager.cc:4841] TS ce9cf74d6e734b2ab8dfe1044c21e1d3 (127.22.157.195:41605): tablet e9e2407ebf82420881eb5e96cb2aaf49 (table test-table [id=4e2da24033a04066a166e5305a2cac13]) successfully deleted
I20260928 15:54:25.482088 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 6478
I20260928 15:54:25.502413 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 6612
Sep 28 15:54:25 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:54:25.520787 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.503253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.22.157.194:37431)
0928 15:54:25.503418 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.503422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.503440 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:54:25.503679 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.503683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.503751 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.503862 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:25.505564 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.506385 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.508438 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.508949 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.509088 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.514281 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:25.514294 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:25.514309 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:25.514351 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:25.517646 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.518571 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.518578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.518580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.518703 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.519100 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.519104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.519106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.519527 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:25.519858 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.520144 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.520371 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:54:25.521507 6735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:25.615810 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 6612
I20260928 15:54:25.635263 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 6746
Sep 28 15:54:25 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:54:25.653324 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:25.636432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.22.157.195:41605)
0928 15:54:25.636659 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:25.636664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:25.636683 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:54:25.636791 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:25.636795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:25.636862 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:25.636954 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:25.638408 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.639202 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.640882 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:25.641308 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:25.641421 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:25.646781 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:25.646799 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:25.646816 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:25.646868 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:25.650145 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.650975 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.650984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.650986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.651156 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:25.651514 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:25.651518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:25.651520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:25.651985 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:25.652224 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:25.652639 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 15:54:25.653056 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 15:54:25.654273 6869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:25.745293 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 6746
I20260928 15:54:25.762750 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 6385
I20260928 15:54:25.764015 6447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:25.882550 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 6385
I20260928 15:54:45.905887 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45363
--webserver_interface=127.22.157.254
--webserver_port=42989
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:45363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:54:46.258921 6924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:46.259230 6924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:46.259321 6924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:46.267036 6924 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:54:46.267158 6924 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:54:46.267197 6924 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:54:46.267242 6924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:54:46.267298 6924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:46.267369 6924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:46.267413 6924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:54:46.267468 6924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:54:46.277812 6924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:45363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:46.279165 6924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:46.280493 6924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:46.305724 6930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:46.305685 6933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:46.305650 6931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:46.306631 6924 server_base.cc:1034] running on GCE node
I20260928 15:54:46.307507 6924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:46.309295 6924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:46.310576 6924 hybrid_clock.cc:648] HybridClock initialized: now 1790610886310496 us; error 45 us; skew 500 ppm
I20260928 15:54:46.311031 6924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:46 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610886, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:46.321313 6924 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:54:46.323330 6924 webserver.cc:460] Webserver started at http://127.22.157.254:42989/ using document root <none> and password file <none>
I20260928 15:54:46.324234 6924 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:46.324345 6924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:46.330825 6924 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:54:46.334259 6940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:46.336110 6924 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:54:46.336355 6924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "cd396c9cdade480da35a6e027dc4ae32"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "e7de0dbd095fc8ac16ca70c52bed49e6"
server_key_iv: "ce5083d554bd4fb9e2926235da31ee97"
server_key_version: "encryptionkey@0"
I20260928 15:54:46.337721 6924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:46.368119 6924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:46.373088 6924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:46.373433 6924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:46.394073 6924 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:45363
I20260928 15:54:46.394205 6991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:45363 every 8 connection(s)
I20260928 15:54:46.395745 6924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:54:46.403744 6992 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:54:46.404915 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 6924
I20260928 15:54:46.405773 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:42881
--local_ip_for_outbound_sockets=127.22.157.193
--tserver_master_addrs=127.22.157.254:45363
--webserver_port=34215
--webserver_interface=127.22.157.193
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:54:46.407773 6992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Bootstrap starting.
I20260928 15:54:46.430451 6992 log.cc:824] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:46.444986 6992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:54:46.445644 6992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Bootstrap complete.
I20260928 15:54:46.456532 6992 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:54:46.457784 6992 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd396c9cdade480da35a6e027dc4ae32, State: Initialized, Role: FOLLOWER
I20260928 15:54:46.458642 6992 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:54:46.458964 6992 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:54:46.459132 6992 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:54:46.459339 6992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 1 FOLLOWER]: Advancing to term 2
I20260928 15:54:46.464306 6992 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:54:46.465054 6992 leader_election.cc:302] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd396c9cdade480da35a6e027dc4ae32; no voters:
I20260928 15:54:46.465847 6992 leader_election.cc:288] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 15:54:46.466006 6996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 15:54:46.467417 6996 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 2 LEADER]: Becoming Leader. State: Replica: cd396c9cdade480da35a6e027dc4ae32, State: Running, Role: LEADER
I20260928 15:54:46.468101 6996 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:54:46.468941 6992 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:54:46.470789 6998 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd396c9cdade480da35a6e027dc4ae32. Latest consensus state: current_term: 2 leader_uuid: "cd396c9cdade480da35a6e027dc4ae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } } }
I20260928 15:54:46.471208 6998 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: This master's current role is: LEADER
I20260928 15:54:46.471796 6997 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cd396c9cdade480da35a6e027dc4ae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } } }
I20260928 15:54:46.472095 6997 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: This master's current role is: LEADER
I20260928 15:54:46.480386 7001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:54:46.491905 7001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:54:46.494118 7001 catalog_manager.cc:1259] Loaded cluster ID: c473759299bc4877840826938e0d9682
I20260928 15:54:46.494295 7001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:54:46.500216 7001 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:54:46.502663 7001 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Loaded TSK: 0
I20260928 15:54:46.510242 7013 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 15:54:46.531625 7001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Generated new TSK 1
I20260928 15:54:46.532043 7001 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 15:54:46.797667 6994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:46.798038 6994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:46.798211 6994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:46.806326 6994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:46.806516 6994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:46.806701 6994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:54:46.817322 6994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:42881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=34215
--tserver_master_addrs=127.22.157.254:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:46.818843 6994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:46.820305 6994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:46.837842 7020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:46.837901 7022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:46.838157 7019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:46.838624 6994 server_base.cc:1034] running on GCE node
I20260928 15:54:46.839366 6994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:46.840469 6994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:46.841687 6994 hybrid_clock.cc:648] HybridClock initialized: now 1790610886841602 us; error 79 us; skew 500 ppm
I20260928 15:54:46.842038 6994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:46 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610886, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:46.845758 6994 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:54:46.847523 6994 webserver.cc:460] Webserver started at http://127.22.157.193:34215/ using document root <none> and password file <none>
I20260928 15:54:46.848258 6994 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:46.848421 6994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:46.854171 6994 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:54:46.857019 7029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:46.858523 6994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:54:46.858714 6994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "a785046b3220198b52ac497d54b3dfa1"
server_key_iv: "e2194f4274e1d442f6781c954b001fc7"
server_key_version: "encryptionkey@0"
I20260928 15:54:46.859725 6994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:46.882056 6994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:46.886551 6994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:46.886843 6994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:46.887986 6994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:46.889516 6994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:46.889613 6994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:46.889778 6994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:46.889878 6994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:46.926980 6994 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:42881
I20260928 15:54:46.927075 7141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:42881 every 8 connection(s)
I20260928 15:54:46.928480 6994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:54:46.934749 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 6994
I20260928 15:54:46.935580 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:37431
--local_ip_for_outbound_sockets=127.22.157.194
--tserver_master_addrs=127.22.157.254:45363
--webserver_port=33003
--webserver_interface=127.22.157.194
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:54:46 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610886, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:46.954355 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:46.931439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:37143 (local address 127.22.157.254:45363)
0928 15:54:46.932185 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:46.932203 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:46.932642 (+ 439us) server_negotiation.cc:407] Connection header received
0928 15:54:46.934926 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:46.934945 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:46.935452 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:46.935926 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:46.937675 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:46.939752 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:46.941543 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:46.942674 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:46.942876 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:46.946508 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:46.946516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:46.946532 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:46.946571 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:46.949582 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:46.950371 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:46.950378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:46.950384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:46.950549 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:46.951003 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:46.951007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:46.951009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:46.951504 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:46.951717 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:46.952527 (+ 810us) server_negotiation.cc:299] Negotiation successful
0928 15:54:46.952950 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":302,"threads_started":1}
I20260928 15:54:46.954352 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:46.931551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:37143)
0928 15:54:46.932464 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:46.932503 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:46.934292 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:46.936191 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:46.936199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:46.936812 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:54:46.937415 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:46.937433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:46.939941 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:46.939945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:46.941334 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:46.941347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:46.942903 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:46.942909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:46.943099 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:46.943833 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:54:46.943862 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:54:46.946192 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:54:46.949741 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:46.949751 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:46.949767 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:46.950212 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:46.950671 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:46.950676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:46.950680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:46.950874 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:46.951656 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:54:46.951665 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:54:46.952022 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 15:54:46.952296 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 15:54:46.952929 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":355,"threads_started":1}
I20260928 15:54:46.958139 7142 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45363
I20260928 15:54:46.958545 7142 heartbeater.cc:464] Registering TS with master...
I20260928 15:54:46.959635 7142 heartbeater.cc:511] Master 127.22.157.254:45363 requested a full tablet report, sending...
I20260928 15:54:46.962416 6957 ts_manager.cc:194] Registered new tserver with Master: 0f3bd8aa63f84b2d9e6d0e4e7a9d17df (127.22.157.193:42881)
I20260928 15:54:46.964272 6957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:37143
W20260928 15:54:47.288911 7146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:47.289209 7146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:47.289306 7146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:47.302871 7146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:47.303020 7146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:47.303243 7146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:54:47.321308 7146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:37431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=33003
--tserver_master_addrs=127.22.157.254:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:47.323184 7146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:47.325125 7146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:47.343269 7151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:47.343236 7152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:47.343292 7154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:47.344115 7146 server_base.cc:1034] running on GCE node
I20260928 15:54:47.344913 7146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:47.345913 7146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:47.347123 7146 hybrid_clock.cc:648] HybridClock initialized: now 1790610887347071 us; error 40 us; skew 500 ppm
I20260928 15:54:47.347493 7146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:47 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610887, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:47.350936 7146 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:54:47.352502 7146 webserver.cc:460] Webserver started at http://127.22.157.194:33003/ using document root <none> and password file <none>
I20260928 15:54:47.353451 7146 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:47.353619 7146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:47.359503 7146 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:54:47.362802 7161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:47.364531 7146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:54:47.364815 7146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "15dedc1a851244309b98974faeb361b2"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "0f2ea0cade9af2af5daf1b3a0e852239"
server_key_iv: "f1ea2a7f00a62833212f51ffe4f921ff"
server_key_version: "encryptionkey@0"
I20260928 15:54:47.365981 7146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:47.383414 7146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:47.387869 7146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:47.388185 7146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:47.389297 7146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:47.390739 7146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:47.390872 7146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:47.391013 7146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:47.391100 7146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:47.427887 7146 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:37431
I20260928 15:54:47.427978 7273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:37431 every 8 connection(s)
I20260928 15:54:47.429556 7146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:54:47.435137 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 7146
I20260928 15:54:47.435819 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:41605
--local_ip_for_outbound_sockets=127.22.157.195
--tserver_master_addrs=127.22.157.254:45363
--webserver_port=36021
--webserver_interface=127.22.157.195
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 15:54:47 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610887, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:47.454631 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:47.432377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:36171 (local address 127.22.157.254:45363)
0928 15:54:47.432566 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:47.432571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:47.433820 (+ 1249us) server_negotiation.cc:407] Connection header received
0928 15:54:47.435798 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:47.435804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:47.435935 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:47.436061 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:47.438758 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.439482 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.442197 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.442763 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.442865 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:47.447641 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:47.447654 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:47.447657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:47.447698 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:47.450549 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:47.451794 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:47.451799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:47.451801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:47.451963 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:47.452567 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:47.452572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:47.452574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:47.452981 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:47.453107 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:47.454132 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0928 15:54:47.454370 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:54:47.456470 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:47.432932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:36171)
0928 15:54:47.433639 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:47.433689 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:47.435467 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:47.436418 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:47.436434 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:47.437468 (+ 1034us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:54:47.438507 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:47.438541 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.439671 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.439679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:47.442016 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:47.442037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.443015 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.443025 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:47.443375 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:47.444178 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:54:47.444215 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:54:47.447304 (+ 3089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:54:47.450765 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:47.450779 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:47.450802 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:47.451582 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:47.452094 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:47.452103 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:47.452109 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:47.452411 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:47.453130 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:54:47.453150 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:54:47.453691 (+ 541us) client_negotiation.cc:769] Sending connection context
0928 15:54:47.454035 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 15:54:47.454749 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":272,"threads_started":1}
I20260928 15:54:47.458225 7274 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45363
I20260928 15:54:47.458597 7274 heartbeater.cc:464] Registering TS with master...
I20260928 15:54:47.459555 7274 heartbeater.cc:511] Master 127.22.157.254:45363 requested a full tablet report, sending...
I20260928 15:54:47.461427 6957 ts_manager.cc:194] Registered new tserver with Master: 15dedc1a851244309b98974faeb361b2 (127.22.157.194:37431)
I20260928 15:54:47.462303 6957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:36171
W20260928 15:54:47.791635 7277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:54:47.791935 7277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:54:47.792088 7277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:54:47.799741 7277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:54:47.799871 7277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:54:47.800060 7277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:54:47.810482 7277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:41605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=36021
--tserver_master_addrs=127.22.157.254:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:54:47.811896 7277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:54:47.813329 7277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:54:47.828455 7282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:47.828402 7283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:54:47.828500 7285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:54:47.829725 7277 server_base.cc:1034] running on GCE node
I20260928 15:54:47.830502 7277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:54:47.831576 7277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:54:47.832885 7277 hybrid_clock.cc:648] HybridClock initialized: now 1790610887832798 us; error 63 us; skew 500 ppm
I20260928 15:54:47.833221 7277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:54:47 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610887, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:54:47.836699 7277 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:54:47.838291 7277 webserver.cc:460] Webserver started at http://127.22.157.195:36021/ using document root <none> and password file <none>
I20260928 15:54:47.839041 7277 fs_manager.cc:359] Metadata directory not provided
I20260928 15:54:47.839210 7277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:54:47.845455 7277 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:54:47.848469 7292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:47.850157 7277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:54:47.850378 7277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "be1a624e8a266536c6afe0f4cb893f20"
server_key_iv: "4d8630bcf9c86c5bb014368dd3401b18"
server_key_version: "encryptionkey@0"
I20260928 15:54:47.851521 7277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:54:47.868927 7277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:54:47.874272 7277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:54:47.874585 7277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:54:47.875676 7277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:54:47.877211 7277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:54:47.877305 7277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:47.877475 7277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:54:47.877569 7277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:54:47.912393 7277 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:41605
I20260928 15:54:47.912485 7404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:41605 every 8 connection(s)
I20260928 15:54:47.913996 7277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:54:47.919473 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 7277
Sep 28 15:54:47 dist-test-slave-1vw0 krb5kdc[6372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610887, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:54:47.938602 7409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:47.920528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.22.157.254:45363)
0928 15:54:47.921477 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:47.921481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:47.921498 (+ 17us) server_negotiation.cc:407] Connection header received
0928 15:54:47.921544 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:47.921548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:47.921599 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:47.921690 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:47.923861 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.924989 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.926990 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.927864 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.928000 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:47.932179 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:47.932186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:47.932190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:47.932233 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:47.935376 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:47.936336 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:47.936342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:47.936345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:47.936513 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:47.936990 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:47.936995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:47.936998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:47.937386 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:47.937690 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:47.938005 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 15:54:47.938194 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":843,"thread_start_us":753,"threads_started":1}
I20260928 15:54:47.940111 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:47.916697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:39641 (local address 127.22.157.254:45363)
0928 15:54:47.916928 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:47.916933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:47.918186 (+ 1253us) server_negotiation.cc:407] Connection header received
0928 15:54:47.920689 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:47.920694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:47.920849 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:47.920959 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:47.923548 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.924716 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.927246 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.927791 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.927935 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:47.932707 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:47.932712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:47.932715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:47.932798 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:47.936224 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:47.937276 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:47.937280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:47.937283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:47.937396 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:47.937998 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:47.938003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:47.938006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:47.938380 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:47.938512 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:47.939493 (+ 981us) server_negotiation.cc:299] Negotiation successful
0928 15:54:47.939743 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":105}
I20260928 15:54:47.941293 7407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:47.917263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:39641)
0928 15:54:47.917987 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:47.918033 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:47.920363 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:47.921430 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:47.921447 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:47.922311 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:54:47.923307 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:47.923339 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.924915 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.924919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:47.927089 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:47.927104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.928039 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.928049 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:47.928372 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:47.929350 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:54:47.929385 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:54:47.932482 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:54:47.936404 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:47.936416 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:47.936438 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:47.937154 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:47.937553 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:54:47.937558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:54:47.937562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:54:47.937856 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:54:47.938586 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:54:47.938593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:54:47.939085 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 15:54:47.939398 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 15:54:47.939995 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":311,"threads_started":1}
I20260928 15:54:47.942741 7405 heartbeater.cc:347] Connected to a master server at 127.22.157.254:45363
I20260928 15:54:47.943084 7405 heartbeater.cc:464] Registering TS with master...
I20260928 15:54:47.943849 7405 heartbeater.cc:511] Master 127.22.157.254:45363 requested a full tablet report, sending...
I20260928 15:54:47.945430 6957 ts_manager.cc:194] Registered new tserver with Master: ce9cf74d6e734b2ab8dfe1044c21e1d3 (127.22.157.195:41605)
I20260928 15:54:47.946316 6957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:39641
I20260928 15:54:47.953964 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:54:47.960914 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:47.955052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.22.157.254:45363)
0928 15:54:47.955323 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:47.955336 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:47.955699 (+ 363us) server_negotiation.cc:407] Connection header received
0928 15:54:47.955921 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:47.955925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:47.956070 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:47.956217 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:54:47.956622 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.957538 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.959053 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.959476 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.959572 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:47.960024 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:54:47.960305 (+ 281us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 15:54:47.960601 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42480: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 15:54:47.961027 7144 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42480: token expired
W20260928 15:54:47.961709 6880 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.157.254:45363)
I20260928 15:54:47.967913 7142 heartbeater.cc:503] Master 127.22.157.254:45363 was elected leader, sending a full tablet report...
I20260928 15:54:47.976455 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:47.962234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.22.157.254:45363)
0928 15:54:47.962418 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:47.962425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:47.962460 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:54:47.962630 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:47.962635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:47.962709 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:47.962813 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:47.964482 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.965319 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.966745 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:47.967149 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:47.967288 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:47.970466 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:47.970471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:47.970474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:47.970506 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:47.973095 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:47.973912 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:47.973917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:47.973920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:47.974112 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:47.974614 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:47.974620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:47.974623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:47.975162 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:47.975498 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:47.975821 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 15:54:47.976082 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:54:47.980329 6957 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:54:47.988469 6957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:54:48.027872 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.018423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:55110)
0928 15:54:48.019982 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.020002 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:48.020114 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:48.020782 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:48.020787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:48.020813 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:48.021159 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.021167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.023180 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.023187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.025233 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.025249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.027107 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.027119 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.027293 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.027322 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 15:54:48.027408 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 15:54:48.027536 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1411,"thread_start_us":1784,"threads_started":1}
I20260928 15:54:48.028079 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.001436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.22.157.194:37431)
0928 15:54:48.017189 (+ 15753us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:48.017198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:48.018055 (+ 857us) server_negotiation.cc:407] Connection header received
0928 15:54:48.018172 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:48.018177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:48.018485 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:48.018701 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:48.020760 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.022375 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.025050 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.026622 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.026818 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.027000 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:48.027614 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 15:54:48.027771 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15593,"spinlock_wait_cycles":29056,"thread_start_us":15571,"threads_started":1}
I20260928 15:54:48.027915 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.001377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:41605 (local address 127.0.0.1:39688)
0928 15:54:48.017649 (+ 16272us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.017699 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:48.017909 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:48.018789 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:48.018801 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:48.018846 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:48.019372 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.019387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.021653 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.021659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.024454 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.024476 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.026677 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.026684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.026897 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.026986 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 15:54:48.027150 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 15:54:48.027366 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15904,"mutex_wait_us":2,"thread_start_us":15833,"threads_started":1}
I20260928 15:54:48.028571 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.002139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.22.157.195:41605)
0928 15:54:48.016983 (+ 14844us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:48.016994 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:48.018127 (+ 1133us) server_negotiation.cc:407] Connection header received
0928 15:54:48.018251 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:48.018259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:48.018482 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:48.019588 (+ 1106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:48.019881 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.021502 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.024650 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.026437 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.026664 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.026858 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:48.028017 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0928 15:54:48.028195 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14604,"thread_start_us":14390,"threads_started":1}
I20260928 15:54:48.028445 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.018603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.22.157.193:42881)
0928 15:54:48.020117 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:48.020126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:48.020156 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:54:48.020262 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:48.020272 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:48.020517 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:48.020742 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:48.021713 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.023044 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.025414 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.026857 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.027058 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.027230 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:48.027489 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 15:54:48.027679 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1332,"thread_start_us":732,"threads_started":1}
I20260928 15:54:48.028927 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.001430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:37431 (local address 127.0.0.1:55534)
0928 15:54:48.017751 (+ 16321us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.017771 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:48.017921 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:48.019007 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:48.019015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:48.019071 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:48.020553 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.020569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.022522 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.022527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.024899 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.024911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.027199 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.027206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.027399 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.027435 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:54:48.028341 (+ 906us) client_negotiation.cc:240] Negotiation successful
0928 15:54:48.028494 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16184,"mutex_wait_us":1082,"spinlock_wait_cycles":45696,"thread_start_us":16022,"threads_started":1}
I20260928 15:54:48.032311 7209 tablet_service.cc:1467] Processing CreateTablet for tablet 387cd1d0aad5468e8fae0848a4b82377 (DEFAULT_TABLE table=test-table [id=e6c28d877ae9481891edebed36963497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:54:48.032512 7340 tablet_service.cc:1467] Processing CreateTablet for tablet 387cd1d0aad5468e8fae0848a4b82377 (DEFAULT_TABLE table=test-table [id=e6c28d877ae9481891edebed36963497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:54:48.032512 7077 tablet_service.cc:1467] Processing CreateTablet for tablet 387cd1d0aad5468e8fae0848a4b82377 (DEFAULT_TABLE table=test-table [id=e6c28d877ae9481891edebed36963497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:54:48.033928 7209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387cd1d0aad5468e8fae0848a4b82377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:48.033946 7340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387cd1d0aad5468e8fae0848a4b82377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:48.033972 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 387cd1d0aad5468e8fae0848a4b82377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:54:48.048339 7421 tablet_bootstrap.cc:492] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: Bootstrap starting.
I20260928 15:54:48.050711 7423 tablet_bootstrap.cc:492] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Bootstrap starting.
I20260928 15:54:48.052438 7421 tablet_bootstrap.cc:654] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:48.053452 7422 tablet_bootstrap.cc:492] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Bootstrap starting.
I20260928 15:54:48.054663 7423 tablet_bootstrap.cc:654] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:48.055456 7421 log.cc:824] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:48.055954 7423 log.cc:824] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:48.057009 7422 tablet_bootstrap.cc:654] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Neither blocks nor log segments found. Creating new log.
I20260928 15:54:48.058300 7422 log.cc:824] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Log is configured to *not* fsync() on all Append() calls
I20260928 15:54:48.058794 7421 tablet_bootstrap.cc:492] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: No bootstrap required, opened a new log
I20260928 15:54:48.059127 7421 ts_tablet_manager.cc:1397] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260928 15:54:48.059252 7423 tablet_bootstrap.cc:492] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: No bootstrap required, opened a new log
I20260928 15:54:48.059458 7423 ts_tablet_manager.cc:1397] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:54:48.061722 7422 tablet_bootstrap.cc:492] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: No bootstrap required, opened a new log
I20260928 15:54:48.062005 7422 ts_tablet_manager.cc:1397] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 15:54:48.067694 7421 raft_consensus.cc:348] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:48.068162 7421 raft_consensus.cc:374] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:48.068300 7421 raft_consensus.cc:729] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15dedc1a851244309b98974faeb361b2, State: Initialized, Role: FOLLOWER
I20260928 15:54:48.069067 7421 consensus_queue.cc:260] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:48.070434 7274 heartbeater.cc:503] Master 127.22.157.254:45363 was elected leader, sending a full tablet report...
I20260928 15:54:48.070588 7423 raft_consensus.cc:348] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:48.071111 7423 raft_consensus.cc:374] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:48.071223 7423 raft_consensus.cc:729] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9cf74d6e734b2ab8dfe1044c21e1d3, State: Initialized, Role: FOLLOWER
I20260928 15:54:48.071312 7421 ts_tablet_manager.cc:1428] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 15:54:48.071956 7423 consensus_queue.cc:260] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:48.073164 7405 heartbeater.cc:503] Master 127.22.157.254:45363 was elected leader, sending a full tablet report...
I20260928 15:54:48.073868 7423 ts_tablet_manager.cc:1428] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
I20260928 15:54:48.072935 7422 raft_consensus.cc:348] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:48.074887 7422 raft_consensus.cc:374] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:54:48.075004 7422 raft_consensus.cc:729] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3bd8aa63f84b2d9e6d0e4e7a9d17df, State: Initialized, Role: FOLLOWER
I20260928 15:54:48.075627 7422 consensus_queue.cc:260] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:48.078089 7422 ts_tablet_manager.cc:1428] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Time spent starting tablet: real 0.016s user 0.005s sys 0.009s
W20260928 15:54:48.165779 7406 tablet.cc:2369] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:54:48.181138 7143 tablet.cc:2369] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:54:48.181689 7275 tablet.cc:2369] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:54:48.283571 7427 raft_consensus.cc:482] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:54:48.283946 7427 raft_consensus.cc:504] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:48.285828 7427 leader_election.cc:288] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce9cf74d6e734b2ab8dfe1044c21e1d3 (127.22.157.195:41605), 0f3bd8aa63f84b2d9e6d0e4e7a9d17df (127.22.157.193:42881)
I20260928 15:54:48.293249 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.286168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:45489 (local address 127.22.157.195:41605)
0928 15:54:48.286397 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:48.286406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:48.287417 (+ 1011us) server_negotiation.cc:407] Connection header received
0928 15:54:48.287714 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:48.287723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:48.287805 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:48.287875 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:48.288541 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.289384 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.291345 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.292178 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.292355 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.292452 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:48.292809 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 15:54:48.292954 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:54:48.293313 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.286443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:60917 (local address 127.22.157.193:42881)
0928 15:54:48.286725 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:48.286732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:48.287567 (+ 835us) server_negotiation.cc:407] Connection header received
0928 15:54:48.287762 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:48.287766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:48.287827 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:48.287963 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:54:48.288710 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.289519 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.291397 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.292189 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.292385 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.292461 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:48.292914 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 15:54:48.293063 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:54:48.293392 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.286257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:41605 (local address 127.22.157.194:45489)
0928 15:54:48.287282 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.287303 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:48.287500 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:48.287927 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:48.287934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:48.287962 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:48.288377 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.288390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.289526 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.289531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.291182 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.291195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.292419 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.292425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.292592 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.292634 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:54:48.292770 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:54:48.292933 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":82,"thread_start_us":578,"threads_started":1}
I20260928 15:54:48.293852 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.286423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.22.157.194:60917)
0928 15:54:48.287456 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.287471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:54:48.287615 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:54:48.287984 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:54:48.287988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:54:48.288017 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:54:48.288526 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.288538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.289664 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.289668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.291204 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:54:48.291217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.292454 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.292460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:54:48.292652 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.292677 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 15:54:48.292857 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 15:54:48.292985 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":67,"thread_start_us":632,"threads_started":1}
I20260928 15:54:48.294560 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387cd1d0aad5468e8fae0848a4b82377" candidate_uuid: "15dedc1a851244309b98974faeb361b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" is_pre_election: true
I20260928 15:54:48.294909 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387cd1d0aad5468e8fae0848a4b82377" candidate_uuid: "15dedc1a851244309b98974faeb361b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" is_pre_election: true
I20260928 15:54:48.295253 7360 raft_consensus.cc:2454] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15dedc1a851244309b98974faeb361b2 in term 0.
I20260928 15:54:48.295527 7097 raft_consensus.cc:2454] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15dedc1a851244309b98974faeb361b2 in term 0.
I20260928 15:54:48.296109 7165 leader_election.cc:302] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15dedc1a851244309b98974faeb361b2, ce9cf74d6e734b2ab8dfe1044c21e1d3; no voters:
I20260928 15:54:48.296489 7427 raft_consensus.cc:2781] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:54:48.296686 7427 raft_consensus.cc:482] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:54:48.296864 7427 raft_consensus.cc:3037] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:48.299427 7427 raft_consensus.cc:504] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:48.300053 7427 leader_election.cc:288] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [CANDIDATE]: Term 1 election: Requested vote from peers ce9cf74d6e734b2ab8dfe1044c21e1d3 (127.22.157.195:41605), 0f3bd8aa63f84b2d9e6d0e4e7a9d17df (127.22.157.193:42881)
I20260928 15:54:48.300554 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387cd1d0aad5468e8fae0848a4b82377" candidate_uuid: "15dedc1a851244309b98974faeb361b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3"
I20260928 15:54:48.300554 7097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "387cd1d0aad5468e8fae0848a4b82377" candidate_uuid: "15dedc1a851244309b98974faeb361b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df"
I20260928 15:54:48.300797 7360 raft_consensus.cc:3037] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:48.300809 7097 raft_consensus.cc:3037] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:54:48.303084 7097 raft_consensus.cc:2454] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15dedc1a851244309b98974faeb361b2 in term 1.
I20260928 15:54:48.303231 7360 raft_consensus.cc:2454] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15dedc1a851244309b98974faeb361b2 in term 1.
I20260928 15:54:48.303619 7163 leader_election.cc:302] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f3bd8aa63f84b2d9e6d0e4e7a9d17df, 15dedc1a851244309b98974faeb361b2; no voters:
I20260928 15:54:48.303970 7427 raft_consensus.cc:2781] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:54:48.304448 7427 raft_consensus.cc:686] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 1 LEADER]: Becoming Leader. State: Replica: 15dedc1a851244309b98974faeb361b2, State: Running, Role: LEADER
I20260928 15:54:48.305007 7427 consensus_queue.cc:237] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } }
I20260928 15:54:48.309965 6957 catalog_manager.cc:5495] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15dedc1a851244309b98974faeb361b2 (127.22.157.194). New cstate: current_term: 1 leader_uuid: "15dedc1a851244309b98974faeb361b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15dedc1a851244309b98974faeb361b2" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 } health_report { overall_health: UNKNOWN } } }
I20260928 15:54:48.382825 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.376835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.22.157.194:37431)
0928 15:54:48.377067 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:48.377078 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:48.377238 (+ 160us) server_negotiation.cc:407] Connection header received
0928 15:54:48.377442 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:48.377446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:48.377524 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:48.377682 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:54:48.378012 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.378753 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.380393 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.380903 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.381002 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.381587 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:54:48.382181 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:54:48.382295 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:48.382404 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 15:54:48.382477 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:54:48.391803 7360 raft_consensus.cc:1261] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 1 FOLLOWER]: Refusing update from remote peer 15dedc1a851244309b98974faeb361b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:54:48.391803 7097 raft_consensus.cc:1261] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 1 FOLLOWER]: Refusing update from remote peer 15dedc1a851244309b98974faeb361b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:54:48.393038 7432 consensus_queue.cc:1035] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:54:48.393487 7427 consensus_queue.cc:1035] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:54:48.434916 6957 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:54:48.435225 6957 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:54:48.444541 6957 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Sending DeleteTablet for 3 replicas of tablet 387cd1d0aad5468e8fae0848a4b82377
I20260928 15:54:48.446106 7209 tablet_service.cc:1514] Processing DeleteTablet for tablet 387cd1d0aad5468e8fae0848a4b82377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:54:48 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:55534
I20260928 15:54:48.449443 7340 tablet_service.cc:1514] Processing DeleteTablet for tablet 387cd1d0aad5468e8fae0848a4b82377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:54:48 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:39688
I20260928 15:54:48.456007 7446 tablet_replica.cc:330] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: stopping tablet replica
I20260928 15:54:48.456889 7446 raft_consensus.cc:2229] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:54:48.456954 7077 tablet_service.cc:1514] Processing DeleteTablet for tablet 387cd1d0aad5468e8fae0848a4b82377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:54:48 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:55110
I20260928 15:54:48.457693 7446 raft_consensus.cc:2258] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:54:48.461676 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 6994
I20260928 15:54:48.463340 7446 ts_tablet_manager.cc:1905] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:48.470692 7447 tablet_replica.cc:330] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: stopping tablet replica
I20260928 15:54:48.475416 7445 tablet_replica.cc:330] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: stopping tablet replica
I20260928 15:54:48.475935 7445 raft_consensus.cc:2229] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:54:48.476832 7445 raft_consensus.cc:2258] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:54:48.480264 7445 ts_tablet_manager.cc:1905] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:48.487280 7447 raft_consensus.cc:2229] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:54:48.488773 7447 raft_consensus.cc:2258] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:54:48.489626 7446 ts_tablet_manager.cc:1918] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:54:48.489894 7446 log.cc:1196] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/387cd1d0aad5468e8fae0848a4b82377
I20260928 15:54:48.490561 7446 ts_tablet_manager.cc:1939] T 387cd1d0aad5468e8fae0848a4b82377 P ce9cf74d6e734b2ab8dfe1044c21e1d3: Deleting consensus metadata
I20260928 15:54:48.492591 6944 catalog_manager.cc:4841] TS ce9cf74d6e734b2ab8dfe1044c21e1d3 (127.22.157.195:41605): tablet 387cd1d0aad5468e8fae0848a4b82377 (table test-table [id=e6c28d877ae9481891edebed36963497]) successfully deleted
I20260928 15:54:48.494149 7447 ts_tablet_manager.cc:1905] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:54:48.495833 7445 ts_tablet_manager.cc:1918] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:54:48.496048 7445 log.cc:1196] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/387cd1d0aad5468e8fae0848a4b82377
I20260928 15:54:48.496641 7445 ts_tablet_manager.cc:1939] T 387cd1d0aad5468e8fae0848a4b82377 P 15dedc1a851244309b98974faeb361b2: Deleting consensus metadata
I20260928 15:54:48.500237 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.463002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.22.157.193:42881)
0928 15:54:48.463192 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:48.463198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:48.471545 (+ 8347us) server_negotiation.cc:407] Connection header received
0928 15:54:48.471755 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:48.471761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:48.471845 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:48.471957 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:48.474895 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.476132 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.479065 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.479741 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.479864 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.485672 (+ 5808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:48.485695 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:48.485714 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:48.485765 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:48.491610 (+ 5845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:48.497517 (+ 5907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:48.497528 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:48.497532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:48.497734 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:48.498246 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:48.498252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:48.498272 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:48.498808 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:48.499178 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:48.499535 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 15:54:48.499800 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:54:48.501006 7137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:48.500995 6942 catalog_manager.cc:4841] TS 15dedc1a851244309b98974faeb361b2 (127.22.157.194:37431): tablet 387cd1d0aad5468e8fae0848a4b82377 (table test-table [id=e6c28d877ae9481891edebed36963497]) successfully deleted
I20260928 15:54:48.594229 7447 ts_tablet_manager.cc:1918] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:54:48.594591 7447 log.cc:1196] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/387cd1d0aad5468e8fae0848a4b82377
I20260928 15:54:48.595006 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 6994
W20260928 15:54:48.613654 6942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.157.193:42881 (error 108)
I20260928 15:54:48.614288 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 7146
I20260928 15:54:48.616235 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.615465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:55140)
0928 15:54:48.615944 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.616055 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":261,"threads_started":1}
W20260928 15:54:48.616592 6942 catalog_manager.cc:4584] TS 0f3bd8aa63f84b2d9e6d0e4e7a9d17df (127.22.157.193:42881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 387cd1d0aad5468e8fae0848a4b82377: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42881: connect: Connection refused (error 111)
I20260928 15:54:48.631151 7415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.615250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.22.157.194:37431)
0928 15:54:48.615446 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:48.615452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:48.615476 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:54:48.615547 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:48.615552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:48.615640 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:48.615751 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:48.617747 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.618653 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.620904 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.621418 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.621581 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.624572 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:48.624589 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:48.624604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:48.624650 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:48.627927 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:48.628771 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:48.628783 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:48.628790 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:48.628972 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:48.629388 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:48.629393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:48.629397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:48.629863 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:48.630180 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:48.630588 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 15:54:48.630809 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:54:48.631886 7269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:48.670611 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.669896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:55146)
0928 15:54:48.670184 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.670337 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 15:54:48.671739 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.671362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:55154)
0928 15:54:48.671497 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.671570 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 15:54:48.732362 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.731864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:55164)
0928 15:54:48.732109 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.732207 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 15:54:48.733707 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.733268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:55178)
0928 15:54:48.733425 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.733554 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 15:54:48.751451 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 7146
I20260928 15:54:48.774624 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 7277
I20260928 15:54:48.789674 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.775441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.22.157.195:41605)
0928 15:54:48.775626 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:54:48.775633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:54:48.775707 (+ 74us) server_negotiation.cc:407] Connection header received
0928 15:54:48.775921 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:54:48.775925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:54:48.775990 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:54:48.776095 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:54:48.777677 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.778408 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.779909 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:54:48.780320 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:54:48.780465 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:54:48.783515 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:54:48.783531 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:54:48.783547 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:54:48.783585 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:54:48.786492 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:48.787348 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:48.787357 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:48.787359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:48.787518 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:54:48.787967 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:54:48.787972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:54:48.787977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:54:48.788405 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:54:48.788873 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0928 15:54:48.789047 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 15:54:48.789279 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:54:48.790380 7400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:48.814833 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.814307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:55190)
0928 15:54:48.814540 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.814640 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 15:54:48.815990 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:48.815533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:55200)
0928 15:54:48.815754 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:48.815830 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 15:54:48.881587 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 7277
I20260928 15:54:48.899729 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 6924
I20260928 15:54:48.901041 6987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:54:49.017814 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:54:49.017112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42881 (local address 127.0.0.1:55212)
0928 15:54:49.017470 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 15:54:49.017589 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260928 15:54:49.018040 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 6924
I20260928 15:55:09.042549 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45363
--webserver_interface=127.22.157.254
--webserver_port=42989
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:45363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:09.393133 7455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:09.393451 7455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:09.393640 7455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:09.401232 7455 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 15:55:09.401345 7455 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 15:55:09.401432 7455 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 15:55:09.401513 7455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:09.401594 7455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:09.401695 7455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:09.401762 7455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:09.401835 7455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:09.412196 7455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:45363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=42989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:09.413689 7455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:09.415198 7455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:09.428273 7460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:09.428344 7463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:09.428362 7461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:09.428526 7455 server_base.cc:1034] running on GCE node
I20260928 15:55:09.429764 7455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:09.431314 7455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:09.432574 7455 hybrid_clock.cc:648] HybridClock initialized: now 1790610909432490 us; error 62 us; skew 500 ppm
I20260928 15:55:09.433015 7455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:09 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610909, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:09.436432 7455 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:55:09.438066 7455 webserver.cc:460] Webserver started at http://127.22.157.254:42989/ using document root <none> and password file <none>
I20260928 15:55:09.438778 7455 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:09.438874 7455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:09.444977 7455 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:55:09.448063 7470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:09.449888 7455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:09.450138 7455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "cd396c9cdade480da35a6e027dc4ae32"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "e7de0dbd095fc8ac16ca70c52bed49e6"
server_key_iv: "ce5083d554bd4fb9e2926235da31ee97"
server_key_version: "encryptionkey@0"
I20260928 15:55:09.451217 7455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:09.476893 7455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:09.481770 7455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:09.482110 7455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:09.501866 7455 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:45363
I20260928 15:55:09.501915 7521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:45363 every 8 connection(s)
I20260928 15:55:09.503496 7455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:09.511313 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 7455
I20260928 15:55:09.512017 7522 sys_catalog.cc:263] Verifying existing consensus state
I20260928 15:55:09.512317 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:42881
--local_ip_for_outbound_sockets=127.22.157.193
--tserver_master_addrs=127.22.157.254:45363
--webserver_port=34215
--webserver_interface=127.22.157.193
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:55:09.515859 7522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Bootstrap starting.
I20260928 15:55:09.538059 7522 log.cc:824] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:09.556049 7522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 15:55:09.556901 7522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Bootstrap complete.
I20260928 15:55:09.567433 7522 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:55:09.568508 7522 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd396c9cdade480da35a6e027dc4ae32, State: Initialized, Role: FOLLOWER
I20260928 15:55:09.569259 7522 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:55:09.569555 7522 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:09.569707 7522 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:09.569916 7522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 2 FOLLOWER]: Advancing to term 3
I20260928 15:55:09.574510 7522 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:55:09.575111 7522 leader_election.cc:302] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd396c9cdade480da35a6e027dc4ae32; no voters:
I20260928 15:55:09.575891 7522 leader_election.cc:288] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 15:55:09.576053 7526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 15:55:09.577368 7526 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [term 3 LEADER]: Becoming Leader. State: Replica: cd396c9cdade480da35a6e027dc4ae32, State: Running, Role: LEADER
I20260928 15:55:09.578079 7526 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } }
I20260928 15:55:09.578502 7522 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:09.581008 7528 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd396c9cdade480da35a6e027dc4ae32. Latest consensus state: current_term: 3 leader_uuid: "cd396c9cdade480da35a6e027dc4ae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } } }
I20260928 15:55:09.581386 7528 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:09.581817 7527 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cd396c9cdade480da35a6e027dc4ae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd396c9cdade480da35a6e027dc4ae32" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45363 } } }
I20260928 15:55:09.582155 7527 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:09.590857 7532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:09.603446 7532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:09.606209 7532 catalog_manager.cc:1259] Loaded cluster ID: c473759299bc4877840826938e0d9682
I20260928 15:55:09.606480 7532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:09.612829 7532 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:09.616091 7532 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Loaded TSK: 0
I20260928 15:55:09.616678 7532 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd396c9cdade480da35a6e027dc4ae32: Loaded TSK: 1
F20260928 15:55:09.618597 7532 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7eff7cbb09ad google::LogMessage::Fail() at ??:0
@ 0x7eff7cbb388a google::LogMessage::SendToLog() at ??:0
@ 0x7eff7cbb04de google::LogMessage::Flush() at ??:0
@ 0x7eff7cbb1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7eff809147d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7eff8090373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7eff925df1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7eff925de783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7eff9266a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7eff9266a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7eff926b45e9 std::function<>::operator()() at ??:0
@ 0x7eff925e2cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7eff925e0c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7eff92668610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7eff92668439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7eff7f7fc345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7eff7f7ff91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7eff7f7ff799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7eff7f7daeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7eff82ab46db start_thread at ??:0
@ 0x7eff7ba4a61f clone at ??:0
W20260928 15:55:09.965303 7524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:09.965619 7524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:09.965780 7524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:09.973452 7524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:09.973579 7524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:09.973757 7524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:55:09.984126 7524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:42881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=34215
--tserver_master_addrs=127.22.157.254:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:09.986083 7524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:09.987547 7524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:10.001748 7569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:10.001793 7572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:10.001763 7570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:10.003136 7524 server_base.cc:1034] running on GCE node
I20260928 15:55:10.003916 7524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:10.005121 7524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:10.006336 7524 hybrid_clock.cc:648] HybridClock initialized: now 1790610910006237 us; error 88 us; skew 500 ppm
I20260928 15:55:10.006728 7524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:10 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610910, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:10.010365 7524 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:55:10.012166 7524 webserver.cc:460] Webserver started at http://127.22.157.193:34215/ using document root <none> and password file <none>
I20260928 15:55:10.013060 7524 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:10.013235 7524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:10.019197 7524 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 15:55:10.022507 7579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:10.024161 7524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:55:10.024355 7524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "0f3bd8aa63f84b2d9e6d0e4e7a9d17df"
format_stamp: "Formatted at 2026-09-28 15:54:23 on dist-test-slave-1vw0"
server_key: "a785046b3220198b52ac497d54b3dfa1"
server_key_iv: "e2194f4274e1d442f6781c954b001fc7"
server_key_version: "encryptionkey@0"
I20260928 15:55:10.025524 7524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:10.050223 7524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:10.055223 7524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:10.055552 7524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:10.056643 7524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:10.058579 7586 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 15:55:10.064248 7586 ts_tablet_manager.cc:1852] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 15:55:10.065532 7586 ts_tablet_manager.cc:1905] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:10.073158 7586 ts_tablet_manager.cc:1918] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:10.073482 7586 ts_tablet_manager.cc:1939] T 387cd1d0aad5468e8fae0848a4b82377 P 0f3bd8aa63f84b2d9e6d0e4e7a9d17df: Deleting consensus metadata
I20260928 15:55:10.074312 7524 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 15:55:10.074465 7524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260928 15:55:10.074672 7524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:10.074770 7524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:10.112428 7524 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:42881
I20260928 15:55:10.112481 7692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:42881 every 8 connection(s)
I20260928 15:55:10.113824 7524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:55:10.115907 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 7524
I20260928 15:55:10.116611 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:37431
--local_ip_for_outbound_sockets=127.22.157.194
--tserver_master_addrs=127.22.157.254:45363
--webserver_port=33003
--webserver_interface=127.22.157.194
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:55:10.127075 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:10.117371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:48939)
0928 15:55:10.125685 (+ 8314us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:10.125982 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7697,"thread_start_us":7820,"threads_started":1}
W20260928 15:55:10.127902 7693 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:45363 (0 consecutive failures): Network error: Failed to ping master at 127.22.157.254:45363: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
I20260928 15:55:10.129029 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:10.128538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:48375)
0928 15:55:10.128768 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:10.128870 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 15:55:10.130177 7695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:10.129706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:38529)
0928 15:55:10.129833 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:10.129960 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 15:55:10.130545 7693 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:55:10.461645 7696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:10.462049 7696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:10.462209 7696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:10.470000 7696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:10.470183 7696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:10.470379 7696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:55:10.483414 7696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:37431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=33003
--tserver_master_addrs=127.22.157.254:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:10.485052 7696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:10.486567 7696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:10.500936 7701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:10.500909 7704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:10.500886 7702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:10.502261 7696 server_base.cc:1034] running on GCE node
I20260928 15:55:10.502995 7696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:10.504050 7696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:10.505297 7696 hybrid_clock.cc:648] HybridClock initialized: now 1790610910505241 us; error 71 us; skew 500 ppm
I20260928 15:55:10.505638 7696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:10 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610910, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:10.509084 7696 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:55:10.510668 7696 webserver.cc:460] Webserver started at http://127.22.157.194:33003/ using document root <none> and password file <none>
I20260928 15:55:10.511397 7696 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:10.511547 7696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:10.517436 7696 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 15:55:10.520335 7711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:10.522101 7696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:10.522339 7696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "15dedc1a851244309b98974faeb361b2"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "0f2ea0cade9af2af5daf1b3a0e852239"
server_key_iv: "f1ea2a7f00a62833212f51ffe4f921ff"
server_key_version: "encryptionkey@0"
I20260928 15:55:10.523561 7696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:10.552465 7696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:10.557499 7696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:10.557809 7696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:10.558888 7696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:10.560369 7696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:10.560505 7696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:10.560643 7696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:10.560760 7696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:10.597505 7696 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:37431
I20260928 15:55:10.597601 7823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:37431 every 8 connection(s)
I20260928 15:55:10.599062 7696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:55:10.604487 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:10.602382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:53007)
0928 15:55:10.603195 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:10.603480 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":322,"threads_started":1}
W20260928 15:55:10.605265 7824 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:45363 (0 consecutive failures): Network error: Failed to ping master at 127.22.157.254:45363: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
I20260928 15:55:10.606268 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:10.605879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:55541)
0928 15:55:10.606061 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:10.606152 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:55:10.606895 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 7696
I20260928 15:55:10.607528 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:10.607073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:46001)
0928 15:55:10.607243 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:10.607357 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 15:55:10.607776 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:41605
--local_ip_for_outbound_sockets=127.22.157.195
--tserver_master_addrs=127.22.157.254:45363
--webserver_port=36021
--webserver_interface=127.22.157.195
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:10.607864 7824 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 15:55:10.961360 7827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:10.961670 7827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:10.961841 7827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:10.969491 7827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:10.969614 7827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:10.969799 7827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:55:10.980391 7827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:41605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=36021
--tserver_master_addrs=127.22.157.254:45363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:10.981809 7827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:10.983208 7827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:10.996830 7832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:10.996877 7833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:10.996851 7835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:10.997789 7827 server_base.cc:1034] running on GCE node
I20260928 15:55:10.998426 7827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:10.999398 7827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:11.000648 7827 hybrid_clock.cc:648] HybridClock initialized: now 1790610911000592 us; error 39 us; skew 500 ppm
I20260928 15:55:11.001060 7827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:10 dist-test-slave-1vw0 krb5kdc[6372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610910, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:11.004359 7827 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:55:11.006121 7827 webserver.cc:460] Webserver started at http://127.22.157.195:36021/ using document root <none> and password file <none>
I20260928 15:55:11.006855 7827 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:11.006961 7827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:11.012856 7827 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:55:11.016146 7842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:11.017906 7827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:11.018133 7827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "ce9cf74d6e734b2ab8dfe1044c21e1d3"
format_stamp: "Formatted at 2026-09-28 15:54:24 on dist-test-slave-1vw0"
server_key: "be1a624e8a266536c6afe0f4cb893f20"
server_key_iv: "4d8630bcf9c86c5bb014368dd3401b18"
server_key_version: "encryptionkey@0"
I20260928 15:55:11.019302 7827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:11.041239 7827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:11.045745 7827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:11.046075 7827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:11.047127 7827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:11.048548 7827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:11.048679 7827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:11.048836 7827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:11.048928 7827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:11.083662 7827 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:41605
I20260928 15:55:11.083756 7954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:41605 every 8 connection(s)
I20260928 15:55:11.085340 7827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:55:11.087046 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 7827
I20260928 15:55:11.090989 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:11.088946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:53289)
0928 15:55:11.089804 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:11.090090 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":304,"threads_started":1}
W20260928 15:55:11.091601 7955 heartbeater.cc:644] Failed to heartbeat to 127.22.157.254:45363 (0 consecutive failures): Network error: Failed to ping master at 127.22.157.254:45363: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
I20260928 15:55:11.092470 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:11.092124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:37473)
0928 15:55:11.092289 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:11.092364 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 15:55:11.093402 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:11.093061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:34887)
0928 15:55:11.093206 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:11.093290 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:55:11.093700 7955 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 15:55:11.132140 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:11.131304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:38935)
0928 15:55:11.131788 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:11.131878 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":253,"threads_started":1}
I20260928 15:55:11.609494 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:11.608572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:53647)
0928 15:55:11.609145 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:11.609246 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":298,"threads_started":1}
I20260928 15:55:12.095192 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:12.094296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:35991)
0928 15:55:12.094802 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:12.094920 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
I20260928 15:55:12.134184 7962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:12.133320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:56379)
0928 15:55:12.133871 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:12.133961 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":304,"threads_started":1}
I20260928 15:55:12.611568 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:12.610729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:43351)
0928 15:55:12.611220 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:12.611318 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":239,"threads_started":1}
I20260928 15:55:13.097150 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:13.096253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:48021)
0928 15:55:13.096872 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:13.096968 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":252,"threads_started":1}
I20260928 15:55:13.136320 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:13.135284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:56897)
0928 15:55:13.136044 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:13.136152 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":450,"threads_started":1}
I20260928 15:55:13.613729 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:13.612877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:33959)
0928 15:55:13.613378 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:13.613524 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":263,"threads_started":1}
I20260928 15:55:14.099041 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:14.098215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:59211)
0928 15:55:14.098749 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:14.098846 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":289,"threads_started":1}
I20260928 15:55:14.138473 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:14.137528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:48023)
0928 15:55:14.138117 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:14.138243 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":294,"threads_started":1}
I20260928 15:55:14.615911 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:14.614997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:56197)
0928 15:55:14.615612 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:14.615720 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":278,"threads_started":1}
I20260928 15:55:15.101092 7970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:15.100163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:48753)
0928 15:55:15.100691 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:15.100818 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":233,"threads_started":1}
I20260928 15:55:15.140165 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:15.139467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:33959)
0928 15:55:15.139916 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:15.140006 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260928 15:55:15.617820 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:15.617007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:38109)
0928 15:55:15.617542 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:15.617634 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":289,"threads_started":1}
I20260928 15:55:16.102974 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:16.102119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:60775)
0928 15:55:16.102719 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:16.102800 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":316,"threads_started":1}
I20260928 15:55:16.142096 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:16.141267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:56197)
0928 15:55:16.141775 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:16.141870 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":267,"threads_started":1}
I20260928 15:55:16.619781 7975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:16.618970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:40871)
0928 15:55:16.619507 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:16.619611 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":275,"threads_started":1}
I20260928 15:55:17.104673 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:17.103875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:52405)
0928 15:55:17.104354 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:17.104448 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":256,"threads_started":1}
I20260928 15:55:17.144166 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:17.143253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:50679)
0928 15:55:17.143858 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:17.143954 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":329,"threads_started":1}
I20260928 15:55:17.622262 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:17.621026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:51223)
0928 15:55:17.621814 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:17.621925 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":347,"threads_started":1}
I20260928 15:55:18.106601 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:18.105739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:41259)
0928 15:55:18.106295 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:18.106377 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":247,"threads_started":1}
I20260928 15:55:18.146026 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:18.145308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:59377)
0928 15:55:18.145753 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:18.145842 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
I20260928 15:55:18.624308 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:18.623411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:46113)
0928 15:55:18.624033 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:18.624136 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":294,"threads_started":1}
I20260928 15:55:19.108628 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:19.107701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:50329)
0928 15:55:19.108284 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:19.108397 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":316,"threads_started":1}
I20260928 15:55:19.148003 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:19.147159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:45009)
0928 15:55:19.147727 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:19.147830 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":263,"threads_started":1}
I20260928 15:55:19.626147 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:19.625349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:44123)
0928 15:55:19.625850 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:19.625938 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":242,"threads_started":1}
I20260928 15:55:20.110693 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:20.109752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:48693)
0928 15:55:20.110434 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:20.110514 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":366,"threads_started":1}
I20260928 15:55:20.150156 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:20.149316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:42513)
0928 15:55:20.149832 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:20.149935 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":285,"threads_started":1}
I20260928 15:55:20.628046 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:20.627208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:47697)
0928 15:55:20.627786 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:20.627878 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":303,"threads_started":1}
I20260928 15:55:21.112423 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:21.111672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:47881)
0928 15:55:21.112162 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:21.112249 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":264,"threads_started":1}
I20260928 15:55:21.152269 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:21.151274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:36837)
0928 15:55:21.151899 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:21.152037 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":378,"threads_started":1}
I20260928 15:55:21.630033 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:21.629147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:48459)
0928 15:55:21.629683 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:21.629835 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":293,"threads_started":1}
I20260928 15:55:22.114319 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:22.113502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:60969)
0928 15:55:22.114026 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:22.114126 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":282,"threads_started":1}
I20260928 15:55:22.154171 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:22.153346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:39485)
0928 15:55:22.153880 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:22.153981 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":313,"threads_started":1}
I20260928 15:55:22.632035 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:22.631168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:33773)
0928 15:55:22.631746 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:22.631849 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":309,"threads_started":1}
I20260928 15:55:23.116149 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:23.115419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:46115)
0928 15:55:23.115894 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:23.115982 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260928 15:55:23.156016 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:23.155248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:37005)
0928 15:55:23.155726 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:23.155830 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":230,"threads_started":1}
I20260928 15:55:23.634083 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:23.633183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:49403)
0928 15:55:23.633760 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:23.633871 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
I20260928 15:55:24.117969 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:24.117178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:39921)
0928 15:55:24.117686 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:24.117787 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":240,"threads_started":1}
I20260928 15:55:24.157886 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:24.157077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:56163)
0928 15:55:24.157593 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:24.157685 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20260928 15:55:24.635938 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:24.635121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:51299)
0928 15:55:24.635642 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:24.635739 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":255,"threads_started":1}
I20260928 15:55:25.119776 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:25.118972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:35435)
0928 15:55:25.119464 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:25.119592 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":262,"threads_started":1}
I20260928 15:55:25.160151 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:25.159318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:59589)
0928 15:55:25.159843 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:25.159956 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":296,"threads_started":1}
I20260928 15:55:25.637916 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:25.637116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.194:52665)
0928 15:55:25.637605 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:25.637728 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
I20260928 15:55:26.090499 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 7524
I20260928 15:55:26.118081 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:26.091745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.22.157.193:42881)
0928 15:55:26.092500 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:26.092522 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:26.092577 (+ 55us) server_negotiation.cc:407] Connection header received
0928 15:55:26.094665 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:26.094678 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:26.095183 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:26.095510 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:26.097207 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:26.098518 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:26.100008 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:26.100771 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:26.100926 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:26.110599 (+ 9673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:26.110609 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:26.110628 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:26.110684 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:26.114211 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:26.115013 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:26.115023 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:26.115033 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:26.115258 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:26.115684 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:26.115690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:26.115693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:26.116463 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:26.116795 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:26.117290 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 15:55:26.117776 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":348,"threads_started":1}
I20260928 15:55:26.119513 7688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:26.124634 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:26.121092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.195:36289)
0928 15:55:26.124337 (+ 3245us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:26.124439 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3119,"thread_start_us":3008,"threads_started":1}
I20260928 15:55:26.206826 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 7524
I20260928 15:55:26.206895 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:26.205688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:45363 (local address 127.22.157.193:49033)
0928 15:55:26.206285 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:26.206591 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.254:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":279,"threads_started":1}
I20260928 15:55:26.226203 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 7696
I20260928 15:55:26.244961 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:26.227117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.22.157.194:37431)
0928 15:55:26.227725 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:26.227739 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:26.227774 (+ 35us) server_negotiation.cc:407] Connection header received
0928 15:55:26.229476 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:26.229486 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:26.229932 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:26.230307 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:26.231819 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:26.233173 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:26.234453 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:26.235163 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:26.235362 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:26.238142 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:26.238150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:26.238167 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:26.238206 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:26.241283 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:26.242066 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:26.242078 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:26.242095 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:26.242282 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:26.242703 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:26.242708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:26.242711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:26.243286 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:26.243633 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:26.244108 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 15:55:26.244571 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":257,"threads_started":1}
I20260928 15:55:26.246475 7819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:26.333832 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 7696
I20260928 15:55:26.353588 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 7827
I20260928 15:55:26.372617 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:26.354654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.22.157.195:41605)
0928 15:55:26.355307 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:26.355322 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:26.355367 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:55:26.357120 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:26.357129 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:26.357588 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:26.357892 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:26.359350 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:26.360632 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:26.362144 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:26.362965 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:26.363127 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:26.365922 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:26.365930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:26.365946 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:26.365988 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:26.369231 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:26.369998 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:26.370006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:26.370012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:26.370205 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:26.370605 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:26.370608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:26.370611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:26.371278 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:26.371523 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:26.371913 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 15:55:26.372265 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":307,"threads_started":1}
I20260928 15:55:26.374255 7950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:26.463336 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 7827
2026-09-28T15:55:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65116 ms)
[----------] 30 tests from SecurityITest (307691 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:55:26 dist-test-slave-1vw0 krb5kdc[8013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:55:26 dist-test-slave-1vw0 krb5kdc[8013](info): set up 2 sockets
Sep 28 15:55:26 dist-test-slave-1vw0 krb5kdc[8013](info): commencing operation
krb5kdc: starting...
W20260928 15:55:27.610165 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:55:27 dist-test-slave-1vw0 krb5kdc[8013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:55:27.909314 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:39549
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:39549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:28.257865 8026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:28.258255 8026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:28.258424 8026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:28.266098 8026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:28.266260 8026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:28.266379 8026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:28.266477 8026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:28.266580 8026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:28.277004 8026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:39549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:39549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:28.278472 8026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:28.279863 8026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:28.292606 8032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:28.292626 8034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:28.292706 8031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:28.293457 8026 server_base.cc:1034] running on GCE node
I20260928 15:55:28.294312 8026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:28.295877 8026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:28.297140 8026 hybrid_clock.cc:648] HybridClock initialized: now 1790610928297072 us; error 63 us; skew 500 ppm
I20260928 15:55:28.297511 8026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:28 dist-test-slave-1vw0 krb5kdc[8013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610928, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:28.300920 8026 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:55:28.302834 8026 webserver.cc:460] Webserver started at http://127.22.157.254:37679/ using document root <none> and password file <none>
I20260928 15:55:28.303792 8026 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:28.303972 8026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:28.304407 8026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:28.307704 8026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "01766adef7b140c9a901168e4d004fb8"
format_stamp: "Formatted at 2026-09-28 15:55:28 on dist-test-slave-1vw0"
server_key: "6572e0aac0ddab6bc43c2bbe80e062ca"
server_key_iv: "00f6d0d8d33e2b564f1487a585cd9d8c"
server_key_version: "encryptionkey@0"
I20260928 15:55:28.308534 8026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "01766adef7b140c9a901168e4d004fb8"
format_stamp: "Formatted at 2026-09-28 15:55:28 on dist-test-slave-1vw0"
server_key: "6572e0aac0ddab6bc43c2bbe80e062ca"
server_key_iv: "00f6d0d8d33e2b564f1487a585cd9d8c"
server_key_version: "encryptionkey@0"
I20260928 15:55:28.314420 8026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 15:55:28.318256 8041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:28.319995 8026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:55:28.320245 8026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "01766adef7b140c9a901168e4d004fb8"
format_stamp: "Formatted at 2026-09-28 15:55:28 on dist-test-slave-1vw0"
server_key: "6572e0aac0ddab6bc43c2bbe80e062ca"
server_key_iv: "00f6d0d8d33e2b564f1487a585cd9d8c"
server_key_version: "encryptionkey@0"
I20260928 15:55:28.320498 8026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:28.349788 8026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:28.354636 8026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:28.354956 8026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:28.375280 8026 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:39549
I20260928 15:55:28.375288 8092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:39549 every 8 connection(s)
I20260928 15:55:28.376924 8026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:28.380139 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 8026
I20260928 15:55:28.380419 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:28.380901 8093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:28.381255 23159 external_mini_cluster.cc:1428] Setting key 4f58ca80eaf78141ee160194aaca48e0
I20260928 15:55:28.395107 8093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8: Bootstrap starting.
Sep 28 15:55:28 dist-test-slave-1vw0 krb5kdc[8013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:28.398989 8093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:28.400139 8093 log.cc:824] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:28.403584 8093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8: No bootstrap required, opened a new log
I20260928 15:55:28.407765 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:28.383348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.22.157.254:39549)
0928 15:55:28.384380 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:28.384392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:28.384435 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:55:28.386877 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:28.386904 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:28.387348 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:28.387711 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:28.389667 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:28.391021 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:28.392957 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:28.393662 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:28.393804 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:28.398840 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:28.398848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:28.398864 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:28.398909 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:28.402681 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:28.403447 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:28.403455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:28.403463 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:28.403716 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:28.404158 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:28.404163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:28.404166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:28.404942 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:28.405277 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:28.405731 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 15:55:28.406243 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":504,"threads_started":1}
I20260928 15:55:28.414264 8093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01766adef7b140c9a901168e4d004fb8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39549 } }
I20260928 15:55:28.414716 8093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:28.414786 8093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01766adef7b140c9a901168e4d004fb8, State: Initialized, Role: FOLLOWER
I20260928 15:55:28.415297 8093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01766adef7b140c9a901168e4d004fb8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39549 } }
I20260928 15:55:28.415510 8093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:28.415606 8093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:28.415726 8093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:28.417991 8093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01766adef7b140c9a901168e4d004fb8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39549 } }
I20260928 15:55:28.418457 8093 leader_election.cc:302] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01766adef7b140c9a901168e4d004fb8; no voters:
I20260928 15:55:28.419027 8093 leader_election.cc:288] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:28.419195 8098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:28.419893 8098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [term 1 LEADER]: Becoming Leader. State: Replica: 01766adef7b140c9a901168e4d004fb8, State: Running, Role: LEADER
I20260928 15:55:28.420646 8098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01766adef7b140c9a901168e4d004fb8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39549 } }
I20260928 15:55:28.421531 8093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:28.423408 8099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01766adef7b140c9a901168e4d004fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01766adef7b140c9a901168e4d004fb8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39549 } } }
I20260928 15:55:28.423743 8099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:28.424618 8104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:28.424461 8100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01766adef7b140c9a901168e4d004fb8. Latest consensus state: current_term: 1 leader_uuid: "01766adef7b140c9a901168e4d004fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01766adef7b140c9a901168e4d004fb8" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 39549 } } }
I20260928 15:55:28.426216 8100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:28.430737 8104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:28.440387 8104 catalog_manager.cc:1347] Generated new cluster ID: c1cb9e50314d46818d28355361344698
I20260928 15:55:28.440557 8104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:28.462131 8104 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:28.464150 8104 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:28.474316 8104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8: Generated new TSK 0
I20260928 15:55:28.475271 8104 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:55:28.549213 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39549
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:28.904552 8119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:28.904968 8119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:28.905092 8119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:28.912659 8119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:28.912853 8119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:28.912992 8119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:55:28.923861 8119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:28.925648 8119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:28.927378 8119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:28.941146 8124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:28.941815 8125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:28.942824 8127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:28.943326 8119 server_base.cc:1034] running on GCE node
I20260928 15:55:28.944024 8119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:28.945112 8119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:28.946363 8119 hybrid_clock.cc:648] HybridClock initialized: now 1790610928946287 us; error 71 us; skew 500 ppm
I20260928 15:55:28.946717 8119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:28 dist-test-slave-1vw0 krb5kdc[8013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610928, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:28.950239 8119 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:55:28.951885 8119 webserver.cc:460] Webserver started at http://127.22.157.193:42367/ using document root <none> and password file <none>
I20260928 15:55:28.952658 8119 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:28.952836 8119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:28.953171 8119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:28.956250 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "adc7c978640346f1a9e30d5769a4c4ff"
format_stamp: "Formatted at 2026-09-28 15:55:28 on dist-test-slave-1vw0"
server_key: "9d7ac7a9d1edb6707e5a6cb4aa2d0972"
server_key_iv: "480adea10fb4cc91ebd569e405fa4b38"
server_key_version: "encryptionkey@0"
I20260928 15:55:28.957139 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "adc7c978640346f1a9e30d5769a4c4ff"
format_stamp: "Formatted at 2026-09-28 15:55:28 on dist-test-slave-1vw0"
server_key: "9d7ac7a9d1edb6707e5a6cb4aa2d0972"
server_key_iv: "480adea10fb4cc91ebd569e405fa4b38"
server_key_version: "encryptionkey@0"
I20260928 15:55:28.962968 8119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 15:55:28.966888 8134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:28.968412 8119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 15:55:28.968681 8119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "adc7c978640346f1a9e30d5769a4c4ff"
format_stamp: "Formatted at 2026-09-28 15:55:28 on dist-test-slave-1vw0"
server_key: "9d7ac7a9d1edb6707e5a6cb4aa2d0972"
server_key_iv: "480adea10fb4cc91ebd569e405fa4b38"
server_key_version: "encryptionkey@0"
I20260928 15:55:28.968972 8119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:28.982007 8119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:28.986748 8119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:28.987071 8119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:28.988126 8119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:28.989727 8119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:28.989837 8119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:28.989987 8119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:28.990061 8119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:29.026686 8119 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:42205
I20260928 15:55:29.026726 8246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:42205 every 8 connection(s)
I20260928 15:55:29.028234 8119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:55:29.031020 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 8119
I20260928 15:55:29.031314 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:55:29.032123 23159 external_mini_cluster.cc:1428] Setting key b750ed83fbc79c5a5470469e80072358
Sep 28 15:55:29 dist-test-slave-1vw0 krb5kdc[8013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610928, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:29.054445 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:29.030987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:58201 (local address 127.22.157.254:39549)
0928 15:55:29.031557 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:29.031565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:29.032771 (+ 1206us) server_negotiation.cc:407] Connection header received
0928 15:55:29.035176 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:29.035182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:29.035267 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:29.035379 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:29.037707 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:29.038682 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:29.041301 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:29.041794 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:29.041899 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:29.046537 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:29.046546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:29.046552 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:29.046601 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:29.050565 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:29.051690 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:29.051700 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:29.051706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:29.051875 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:29.052441 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:29.052445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:29.052448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:29.052838 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:29.052963 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:29.053953 (+ 990us) server_negotiation.cc:299] Negotiation successful
0928 15:55:29.054174 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":292,"threads_started":1}
I20260928 15:55:29.056064 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:29.031677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:39549 (local address 127.22.157.193:58201)
0928 15:55:29.032544 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:29.032613 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:29.034811 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:29.035667 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:29.035681 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:29.036559 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:29.037499 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:29.037530 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:29.038852 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:29.038859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:29.041133 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:29.041157 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:29.042049 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:29.042058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:29.042409 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:29.043191 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:55:29.043225 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:29.046234 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:29.050768 (+ 4534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:29.050780 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:29.050803 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:29.051535 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:29.052029 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:29.052036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:29.052042 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:29.052295 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:29.052976 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:29.052988 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:29.053489 (+ 501us) client_negotiation.cc:769] Sending connection context
0928 15:55:29.053827 (+ 338us) client_negotiation.cc:240] Negotiation successful
0928 15:55:29.054463 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":320,"threads_started":1}
I20260928 15:55:29.057730 8247 heartbeater.cc:347] Connected to a master server at 127.22.157.254:39549
I20260928 15:55:29.058111 8247 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:29.058843 8247 heartbeater.cc:511] Master 127.22.157.254:39549 requested a full tablet report, sending...
I20260928 15:55:29.061169 8058 ts_manager.cc:194] Registered new tserver with Master: adc7c978640346f1a9e30d5769a4c4ff (127.22.157.193:42205)
I20260928 15:55:29.062685 8058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:58201
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:55:29.100835 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39549
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:29.448084 8253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:29.448403 8253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:29.448513 8253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:29.455972 8253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:29.456105 8253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:29.456228 8253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:55:29.467211 8253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:29.468549 8253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:29.469898 8253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:29.483952 8261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:29.484012 8258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:29.484038 8259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:29.484771 8253 server_base.cc:1034] running on GCE node
I20260928 15:55:29.485500 8253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:29.486449 8253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:29.487691 8253 hybrid_clock.cc:648] HybridClock initialized: now 1790610929487635 us; error 50 us; skew 500 ppm
I20260928 15:55:29.488041 8253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:29 dist-test-slave-1vw0 krb5kdc[8013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610929, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:29.491341 8253 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:55:29.493027 8253 webserver.cc:460] Webserver started at http://127.22.157.194:34527/ using document root <none> and password file <none>
I20260928 15:55:29.493733 8253 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:29.493881 8253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:29.494222 8253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:29.497278 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "0afe868e9b8d46bab96b64396a870186"
format_stamp: "Formatted at 2026-09-28 15:55:29 on dist-test-slave-1vw0"
server_key: "502bb8cde88a86c62fb8c92191e1d706"
server_key_iv: "52b6699b66ae5b36dab39d997463b1ce"
server_key_version: "encryptionkey@0"
I20260928 15:55:29.498062 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "0afe868e9b8d46bab96b64396a870186"
format_stamp: "Formatted at 2026-09-28 15:55:29 on dist-test-slave-1vw0"
server_key: "502bb8cde88a86c62fb8c92191e1d706"
server_key_iv: "52b6699b66ae5b36dab39d997463b1ce"
server_key_version: "encryptionkey@0"
I20260928 15:55:29.503369 8253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:55:29.507252 8268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:29.508934 8253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:29.509120 8253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "0afe868e9b8d46bab96b64396a870186"
format_stamp: "Formatted at 2026-09-28 15:55:29 on dist-test-slave-1vw0"
server_key: "502bb8cde88a86c62fb8c92191e1d706"
server_key_iv: "52b6699b66ae5b36dab39d997463b1ce"
server_key_version: "encryptionkey@0"
I20260928 15:55:29.509361 8253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:29.538503 8253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:29.543413 8253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:29.543702 8253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:29.544714 8253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:29.546115 8253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:29.546190 8253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:29.546275 8253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:29.546320 8253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:29.581461 8253 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:38405
I20260928 15:55:29.581516 8380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:38405 every 8 connection(s)
I20260928 15:55:29.582846 8253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:55:29.589385 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 8253
I20260928 15:55:29.589615 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:55:29.590083 23159 external_mini_cluster.cc:1428] Setting key 7a0192e7c2a0acec0592e30bbbcbfd2c
Sep 28 15:55:29 dist-test-slave-1vw0 krb5kdc[8013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610929, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:29.604831 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:29.585671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:35153 (local address 127.22.157.254:39549)
0928 15:55:29.586188 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:29.586193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:29.587099 (+ 906us) server_negotiation.cc:407] Connection header received
0928 15:55:29.589096 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:29.589101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:29.589200 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:29.589322 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:29.591188 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:29.592061 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:29.593718 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:29.594154 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:29.594315 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:29.598187 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:29.598193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:29.598196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:29.598232 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:29.601202 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:29.602107 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:29.602112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:29.602114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:29.602255 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:29.602745 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:29.602749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:29.602751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:29.603056 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:29.603227 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:29.604197 (+ 970us) server_negotiation.cc:299] Negotiation successful
0928 15:55:29.604479 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":289,"threads_started":1}
I20260928 15:55:29.606530 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:29.586086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:39549 (local address 127.22.157.194:35153)
0928 15:55:29.586913 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:29.586954 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:29.588793 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:29.589597 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:29.589611 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:29.590251 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:29.590999 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:29.591021 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:29.592190 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:29.592196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:29.593530 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:29.593543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:29.594406 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:29.594414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:29.594670 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:29.595385 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:55:29.595409 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:29.597906 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:29.601395 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:29.601409 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:29.601432 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:29.601944 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:29.602386 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:29.602391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:29.602395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:29.602607 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:29.603216 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:29.603225 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:29.603743 (+ 518us) client_negotiation.cc:769] Sending connection context
0928 15:55:29.604065 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 15:55:29.604717 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":370,"threads_started":1}
I20260928 15:55:29.608304 8381 heartbeater.cc:347] Connected to a master server at 127.22.157.254:39549
I20260928 15:55:29.608676 8381 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:29.609467 8381 heartbeater.cc:511] Master 127.22.157.254:39549 requested a full tablet report, sending...
I20260928 15:55:29.611016 8058 ts_manager.cc:194] Registered new tserver with Master: 0afe868e9b8d46bab96b64396a870186 (127.22.157.194:38405)
I20260928 15:55:29.611876 8058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:35153
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:55:29.656567 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39549
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:29.993011 8387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:29.993335 8387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:29.993513 8387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:30.000970 8387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:30.001149 8387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:30.001336 8387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:55:30.011844 8387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:39549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:30.013505 8387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:30.014971 8387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:30.029047 8392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:30.029105 8393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:30.029310 8395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:30.030532 8387 server_base.cc:1034] running on GCE node
I20260928 15:55:30.031263 8387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:30.032490 8387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:30.033716 8387 hybrid_clock.cc:648] HybridClock initialized: now 1790610930033637 us; error 79 us; skew 500 ppm
I20260928 15:55:30.034000 8387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:30 dist-test-slave-1vw0 krb5kdc[8013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610930, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:30.037238 8387 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:55:30.038738 8387 webserver.cc:460] Webserver started at http://127.22.157.195:36967/ using document root <none> and password file <none>
I20260928 15:55:30.039405 8387 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:30.039507 8387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:30.039758 8387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:30.042758 8387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "3a8544c4c2b7475fa936aacad789d1f5"
format_stamp: "Formatted at 2026-09-28 15:55:30 on dist-test-slave-1vw0"
server_key: "42cadf511807daa51edff40922e08ed1"
server_key_iv: "0cfc49937121172708ca20f15259d30e"
server_key_version: "encryptionkey@0"
I20260928 15:55:30.043538 8387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "3a8544c4c2b7475fa936aacad789d1f5"
format_stamp: "Formatted at 2026-09-28 15:55:30 on dist-test-slave-1vw0"
server_key: "42cadf511807daa51edff40922e08ed1"
server_key_iv: "0cfc49937121172708ca20f15259d30e"
server_key_version: "encryptionkey@0"
I20260928 15:55:30.049013 8387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:55:30.052639 8402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:30.054061 8387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:30.054265 8387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "3a8544c4c2b7475fa936aacad789d1f5"
format_stamp: "Formatted at 2026-09-28 15:55:30 on dist-test-slave-1vw0"
server_key: "42cadf511807daa51edff40922e08ed1"
server_key_iv: "0cfc49937121172708ca20f15259d30e"
server_key_version: "encryptionkey@0"
I20260928 15:55:30.054481 8387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:30.066249 8247 heartbeater.cc:503] Master 127.22.157.254:39549 was elected leader, sending a full tablet report...
I20260928 15:55:30.087311 8387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:30.091883 8387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:30.092199 8387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:30.093235 8387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:30.094697 8387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:30.094805 8387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:30.094964 8387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:30.095044 8387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:30.129458 8387 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:40729
I20260928 15:55:30.129498 8514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:40729 every 8 connection(s)
I20260928 15:55:30.130802 8387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:55:30.135408 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 8387
I20260928 15:55:30.135627 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:55:30.136111 23159 external_mini_cluster.cc:1428] Setting key 68e0f57b322df08f34f5de2308caa4fb
Sep 28 15:55:30 dist-test-slave-1vw0 krb5kdc[8013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610930, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:30.151446 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.133584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:34865 (local address 127.22.157.254:39549)
0928 15:55:30.134093 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.134098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.135118 (+ 1020us) server_negotiation.cc:407] Connection header received
0928 15:55:30.137313 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.137318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.137436 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.137556 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:30.139329 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.140136 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.141772 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.142188 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.142333 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.145888 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:30.145893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:30.145897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:30.145930 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:30.148330 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.149134 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.149146 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.149149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.149270 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.149694 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.149699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.149704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.149967 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:30.150160 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.150820 (+ 660us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.151051 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":249,"threads_started":1}
I20260928 15:55:30.152465 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.134219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:39549 (local address 127.22.157.195:34865)
0928 15:55:30.134895 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:30.134987 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:30.137021 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:30.137819 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:30.137828 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:30.138471 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:30.139148 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.139177 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.140295 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.140308 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.141632 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.141644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.142430 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.142437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.142696 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.143318 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:55:30.143345 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:30.145672 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:30.148474 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:30.148483 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:30.148501 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:30.148997 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:30.149382 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:30.149387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:30.149391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:30.149575 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:30.150103 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:30.150111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:30.150463 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 15:55:30.150734 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 15:55:30.151277 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":283,"threads_started":1}
I20260928 15:55:30.153895 8515 heartbeater.cc:347] Connected to a master server at 127.22.157.254:39549
I20260928 15:55:30.154214 8515 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:30.154932 8515 heartbeater.cc:511] Master 127.22.157.254:39549 requested a full tablet report, sending...
I20260928 15:55:30.156364 8058 ts_manager.cc:194] Registered new tserver with Master: 3a8544c4c2b7475fa936aacad789d1f5 (127.22.157.195:40729)
I20260928 15:55:30.157243 8058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:34865
I20260928 15:55:30.163853 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:55:30.182147 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.168645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.22.157.254:39549)
0928 15:55:30.168904 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.168913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.169385 (+ 472us) server_negotiation.cc:407] Connection header received
0928 15:55:30.169646 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.169650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.169720 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.169828 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:30.171347 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.172115 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.173544 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.173950 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.174091 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.176590 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:30.176595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:30.176598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:30.176639 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:30.179239 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.179916 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.179923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.179927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.180071 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.180505 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.180510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.180513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.180936 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:30.181203 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.181603 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.181835 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:55:30.186280 8058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:55:30.193356 8058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:55:30.227762 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.206773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.22.157.194:38405)
0928 15:55:30.219098 (+ 12325us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.219106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.219128 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:55:30.219205 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.219210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.219451 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.219682 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:30.220998 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.222125 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.225022 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.226467 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.226655 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.226850 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.227210 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.227408 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12168,"thread_start_us":12071,"threads_started":1}
I20260928 15:55:30.227883 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.206522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:38405 (local address 127.0.0.1:58004)
0928 15:55:30.216128 (+ 9606us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:30.216172 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:30.216381 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:30.219970 (+ 3589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:30.219978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:30.220035 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:30.220809 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.220823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.222328 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.222336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.224838 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.224856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.226677 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.226687 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.226904 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.226987 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 15:55:30.227122 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 15:55:30.227382 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9323,"mutex_wait_us":84,"spinlock_wait_cycles":4864,"thread_start_us":10906,"threads_started":1}
I20260928 15:55:30.229348 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.206563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42205 (local address 127.0.0.1:58542)
0928 15:55:30.217736 (+ 11173us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:30.217762 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:30.217919 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:30.220321 (+ 2402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:30.220325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:30.220353 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:30.221084 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.221097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.223143 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.223147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.225786 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.225802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.228278 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.228284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.228527 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.228622 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 15:55:30.228704 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 15:55:30.228916 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10986,"mutex_wait_us":239,"thread_start_us":11624,"threads_started":1}
I20260928 15:55:30.229465 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.208042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.22.157.195:40729)
0928 15:55:30.218992 (+ 10950us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.219013 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.219051 (+ 38us) server_negotiation.cc:407] Connection header received
0928 15:55:30.219162 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.219171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.219452 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.219682 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:30.221222 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.222981 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.226221 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.227699 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.227891 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.228068 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.228927 (+ 859us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.229106 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10744,"thread_start_us":10571,"threads_started":1}
I20260928 15:55:30.229964 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.206966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:40729 (local address 127.0.0.1:43976)
0928 15:55:30.218465 (+ 11499us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:30.218487 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:30.218619 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:30.219970 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:30.219978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:30.220035 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:30.220674 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.220695 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.223142 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.223147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.226060 (+ 2913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.226071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.228264 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.228270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.228532 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.228626 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 15:55:30.228704 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 15:55:30.229710 (+ 1006us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11365,"mutex_wait_us":956,"spinlock_wait_cycles":4480,"thread_start_us":8712,"threads_started":1}
I20260928 15:55:30.230783 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.207189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.22.157.193:42205)
0928 15:55:30.219631 (+ 12442us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.219640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.219671 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:55:30.219770 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.219778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.220060 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.220843 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:30.221247 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.222966 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.225979 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.227654 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.227866 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.228367 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.230410 (+ 2043us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.230558 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12227,"thread_start_us":12074,"threads_started":1}
I20260928 15:55:30.233052 8316 tablet_service.cc:1467] Processing CreateTablet for tablet fdfdfb65f70a4aeaaea0a3fee85fef64 (DEFAULT_TABLE table=test-table [id=f6cb7c52466f4ee88723537ae263e6f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:30.233228 8450 tablet_service.cc:1467] Processing CreateTablet for tablet fdfdfb65f70a4aeaaea0a3fee85fef64 (DEFAULT_TABLE table=test-table [id=f6cb7c52466f4ee88723537ae263e6f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:30.234481 8316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdfdfb65f70a4aeaaea0a3fee85fef64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:30.234629 8450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdfdfb65f70a4aeaaea0a3fee85fef64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:30.237810 8182 tablet_service.cc:1467] Processing CreateTablet for tablet fdfdfb65f70a4aeaaea0a3fee85fef64 (DEFAULT_TABLE table=test-table [id=f6cb7c52466f4ee88723537ae263e6f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:30.239463 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdfdfb65f70a4aeaaea0a3fee85fef64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:30.246385 8534 tablet_bootstrap.cc:492] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: Bootstrap starting.
I20260928 15:55:30.250123 8534 tablet_bootstrap.cc:654] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:30.251197 8534 log.cc:824] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:30.254323 8534 tablet_bootstrap.cc:492] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: No bootstrap required, opened a new log
I20260928 15:55:30.254671 8534 ts_tablet_manager.cc:1397] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260928 15:55:30.255309 8537 tablet_bootstrap.cc:492] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: Bootstrap starting.
I20260928 15:55:30.256999 8535 tablet_bootstrap.cc:492] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: Bootstrap starting.
I20260928 15:55:30.259727 8537 tablet_bootstrap.cc:654] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:30.260494 8535 tablet_bootstrap.cc:654] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:30.261196 8537 log.cc:824] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:30.261626 8535 log.cc:824] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:30.264380 8535 tablet_bootstrap.cc:492] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: No bootstrap required, opened a new log
I20260928 15:55:30.264701 8535 ts_tablet_manager.cc:1397] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260928 15:55:30.266609 8534 raft_consensus.cc:348] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } }
I20260928 15:55:30.267086 8534 raft_consensus.cc:374] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:30.267196 8534 raft_consensus.cc:729] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8544c4c2b7475fa936aacad789d1f5, State: Initialized, Role: FOLLOWER
I20260928 15:55:30.267313 8537 tablet_bootstrap.cc:492] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: No bootstrap required, opened a new log
I20260928 15:55:30.267586 8537 ts_tablet_manager.cc:1397] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260928 15:55:30.267876 8534 consensus_queue.cc:260] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } }
I20260928 15:55:30.273406 8535 raft_consensus.cc:348] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } }
I20260928 15:55:30.273849 8535 raft_consensus.cc:374] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:30.273988 8535 raft_consensus.cc:729] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afe868e9b8d46bab96b64396a870186, State: Initialized, Role: FOLLOWER
I20260928 15:55:30.274626 8535 consensus_queue.cc:260] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } }
I20260928 15:55:30.275092 8515 heartbeater.cc:503] Master 127.22.157.254:39549 was elected leader, sending a full tablet report...
I20260928 15:55:30.276181 8534 ts_tablet_manager.cc:1428] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: Time spent starting tablet: real 0.021s user 0.015s sys 0.007s
I20260928 15:55:30.277100 8381 heartbeater.cc:503] Master 127.22.157.254:39549 was elected leader, sending a full tablet report...
I20260928 15:55:30.277833 8535 ts_tablet_manager.cc:1428] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260928 15:55:30.279997 8537 raft_consensus.cc:348] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } }
I20260928 15:55:30.280478 8537 raft_consensus.cc:374] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:30.280649 8537 raft_consensus.cc:729] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc7c978640346f1a9e30d5769a4c4ff, State: Initialized, Role: FOLLOWER
I20260928 15:55:30.281342 8537 consensus_queue.cc:260] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } }
I20260928 15:55:30.283142 8537 ts_tablet_manager.cc:1428] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
W20260928 15:55:30.334947 8382 tablet.cc:2369] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:55:30.363780 8542 raft_consensus.cc:482] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:55:30.364126 8542 raft_consensus.cc:504] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } }
I20260928 15:55:30.366060 8542 leader_election.cc:288] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afe868e9b8d46bab96b64396a870186 (127.22.157.194:38405), 3a8544c4c2b7475fa936aacad789d1f5 (127.22.157.195:40729)
I20260928 15:55:30.373528 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.366674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:37367 (local address 127.22.157.194:38405)
0928 15:55:30.366868 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.366872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.367798 (+ 926us) server_negotiation.cc:407] Connection header received
0928 15:55:30.368000 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.368004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.368061 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.368162 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:30.368978 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.369691 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.371518 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.372299 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.372499 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.372578 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.372974 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.373123 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 15:55:30.373631 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.366687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:60897 (local address 127.22.157.195:40729)
0928 15:55:30.366942 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.366950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.367700 (+ 750us) server_negotiation.cc:407] Connection header received
0928 15:55:30.367942 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.367946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.368006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.368109 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:30.368850 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.369596 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.371472 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.372383 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.372552 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.372638 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.373247 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.373374 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 15:55:30.374116 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.366673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:38405 (local address 127.22.157.193:37367)
0928 15:55:30.367682 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:30.367697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:30.367853 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:30.368174 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:30.368179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:30.368202 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:30.368806 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.368822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.369842 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.369847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.371353 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.371366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.372510 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.372515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.372702 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.372787 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 15:55:30.372919 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 15:55:30.373070 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":255,"thread_start_us":456,"threads_started":1}
I20260928 15:55:30.374217 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.366668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:40729 (local address 127.22.157.193:60897)
0928 15:55:30.367581 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:30.367597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:30.367784 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:30.368128 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:30.368132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:30.368156 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:30.368636 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.368649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.369727 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.369736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.371316 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:30.371331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.372841 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.372849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:30.373046 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.373079 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:55:30.373213 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 15:55:30.373366 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":24,"thread_start_us":487,"threads_started":1}
I20260928 15:55:30.375231 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdfdfb65f70a4aeaaea0a3fee85fef64" candidate_uuid: "adc7c978640346f1a9e30d5769a4c4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afe868e9b8d46bab96b64396a870186" is_pre_election: true
I20260928 15:55:30.375514 8470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdfdfb65f70a4aeaaea0a3fee85fef64" candidate_uuid: "adc7c978640346f1a9e30d5769a4c4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8544c4c2b7475fa936aacad789d1f5" is_pre_election: true
I20260928 15:55:30.375835 8336 raft_consensus.cc:2454] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc7c978640346f1a9e30d5769a4c4ff in term 0.
I20260928 15:55:30.376080 8470 raft_consensus.cc:2454] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc7c978640346f1a9e30d5769a4c4ff in term 0.
I20260928 15:55:30.376600 8138 leader_election.cc:302] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afe868e9b8d46bab96b64396a870186, adc7c978640346f1a9e30d5769a4c4ff; no voters:
I20260928 15:55:30.377074 8542 raft_consensus.cc:2781] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:55:30.377254 8542 raft_consensus.cc:482] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:55:30.377347 8542 raft_consensus.cc:3037] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:30.380465 8542 raft_consensus.cc:504] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } }
I20260928 15:55:30.381296 8542 leader_election.cc:288] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [CANDIDATE]: Term 1 election: Requested vote from peers 0afe868e9b8d46bab96b64396a870186 (127.22.157.194:38405), 3a8544c4c2b7475fa936aacad789d1f5 (127.22.157.195:40729)
I20260928 15:55:30.381843 8470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdfdfb65f70a4aeaaea0a3fee85fef64" candidate_uuid: "adc7c978640346f1a9e30d5769a4c4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a8544c4c2b7475fa936aacad789d1f5"
I20260928 15:55:30.381829 8336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdfdfb65f70a4aeaaea0a3fee85fef64" candidate_uuid: "adc7c978640346f1a9e30d5769a4c4ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afe868e9b8d46bab96b64396a870186"
I20260928 15:55:30.382066 8336 raft_consensus.cc:3037] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:30.382062 8470 raft_consensus.cc:3037] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [term 0 FOLLOWER]: Advancing to term 1
W20260928 15:55:30.382449 8516 tablet.cc:2369] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:55:30.384819 8470 raft_consensus.cc:2454] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc7c978640346f1a9e30d5769a4c4ff in term 1.
I20260928 15:55:30.385296 8137 leader_election.cc:302] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8544c4c2b7475fa936aacad789d1f5, adc7c978640346f1a9e30d5769a4c4ff; no voters:
I20260928 15:55:30.385311 8336 raft_consensus.cc:2454] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc7c978640346f1a9e30d5769a4c4ff in term 1.
I20260928 15:55:30.385609 8542 raft_consensus.cc:2781] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:30.386154 8542 raft_consensus.cc:686] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 1 LEADER]: Becoming Leader. State: Replica: adc7c978640346f1a9e30d5769a4c4ff, State: Running, Role: LEADER
I20260928 15:55:30.386740 8542 consensus_queue.cc:237] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } }
I20260928 15:55:30.391350 8057 catalog_manager.cc:5495] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff reported cstate change: term changed from 0 to 1, leader changed from <none> to adc7c978640346f1a9e30d5769a4c4ff (127.22.157.193). New cstate: current_term: 1 leader_uuid: "adc7c978640346f1a9e30d5769a4c4ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc7c978640346f1a9e30d5769a4c4ff" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 } health_report { overall_health: UNKNOWN } } }
I20260928 15:55:30.424825 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.418809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58548 (local address 127.22.157.193:42205)
0928 15:55:30.419027 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.419034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.419141 (+ 107us) server_negotiation.cc:407] Connection header received
0928 15:55:30.419401 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.419405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.419498 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.419671 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:55:30.420084 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.420822 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.422378 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.422797 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.422982 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.423571 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:55:30.424166 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:55:30.424339 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.424397 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.424484 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:55:30.433645 8470 raft_consensus.cc:1261] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [term 1 FOLLOWER]: Refusing update from remote peer adc7c978640346f1a9e30d5769a4c4ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:30.433650 8336 raft_consensus.cc:1261] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [term 1 FOLLOWER]: Refusing update from remote peer adc7c978640346f1a9e30d5769a4c4ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:30.434715 8542 consensus_queue.cc:1035] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a8544c4c2b7475fa936aacad789d1f5" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 40729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:30.435029 8545 consensus_queue.cc:1035] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afe868e9b8d46bab96b64396a870186" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 38405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:30.447431 8548 mvcc.cc:205] Tried to move back new op lower bound from 7334342371045675008 to 7334342370868572160. Current Snapshot: MvccSnapshot[applied={T|T < 7334342371045675008}]
I20260928 15:55:30.465415 8057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:30.465731 8057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:30.470615 8057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01766adef7b140c9a901168e4d004fb8: Sending DeleteTablet for 3 replicas of tablet fdfdfb65f70a4aeaaea0a3fee85fef64
I20260928 15:55:30.471921 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet fdfdfb65f70a4aeaaea0a3fee85fef64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:30 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:58542
I20260928 15:55:30.471910 8316 tablet_service.cc:1514] Processing DeleteTablet for tablet fdfdfb65f70a4aeaaea0a3fee85fef64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:30 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:58004
I20260928 15:55:30.472800 8450 tablet_service.cc:1514] Processing DeleteTablet for tablet fdfdfb65f70a4aeaaea0a3fee85fef64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:30 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:43976
I20260928 15:55:30.473101 8558 tablet_replica.cc:330] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: stopping tablet replica
I20260928 15:55:30.473382 8559 tablet_replica.cc:330] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: stopping tablet replica
I20260928 15:55:30.473798 8558 raft_consensus.cc:2229] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:30.473822 8559 raft_consensus.cc:2229] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:55:30.474530 8559 raft_consensus.cc:2258] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:30.474609 8558 raft_consensus.cc:2258] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:30.475224 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 8119
I20260928 15:55:30.477644 8559 ts_tablet_manager.cc:1905] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:30.480252 8558 ts_tablet_manager.cc:1905] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:30.485460 8560 tablet_replica.cc:330] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: stopping tablet replica
I20260928 15:55:30.485956 8560 raft_consensus.cc:2229] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:30.486521 8560 raft_consensus.cc:2258] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:30.492535 8558 ts_tablet_manager.cc:1918] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:30.492797 8558 log.cc:1196] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/fdfdfb65f70a4aeaaea0a3fee85fef64
I20260928 15:55:30.493095 8559 ts_tablet_manager.cc:1918] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:30.493398 8558 ts_tablet_manager.cc:1939] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 0afe868e9b8d46bab96b64396a870186: Deleting consensus metadata
I20260928 15:55:30.493402 8559 log.cc:1196] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/fdfdfb65f70a4aeaaea0a3fee85fef64
I20260928 15:55:30.494041 8559 ts_tablet_manager.cc:1939] T fdfdfb65f70a4aeaaea0a3fee85fef64 P adc7c978640346f1a9e30d5769a4c4ff: Deleting consensus metadata
I20260928 15:55:30.495625 8045 catalog_manager.cc:4841] TS 0afe868e9b8d46bab96b64396a870186 (127.22.157.194:38405): tablet fdfdfb65f70a4aeaaea0a3fee85fef64 (table test-table [id=f6cb7c52466f4ee88723537ae263e6f5]) successfully deleted
I20260928 15:55:30.495795 8042 catalog_manager.cc:4841] TS adc7c978640346f1a9e30d5769a4c4ff (127.22.157.193:42205): tablet fdfdfb65f70a4aeaaea0a3fee85fef64 (table test-table [id=f6cb7c52466f4ee88723537ae263e6f5]) successfully deleted
I20260928 15:55:30.496587 8560 ts_tablet_manager.cc:1905] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:55:30 dist-test-slave-1vw0 krb5kdc[8013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:55:30.504069 8560 ts_tablet_manager.cc:1918] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:30.504266 8560 log.cc:1196] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/fdfdfb65f70a4aeaaea0a3fee85fef64
I20260928 15:55:30.504851 8560 ts_tablet_manager.cc:1939] T fdfdfb65f70a4aeaaea0a3fee85fef64 P 3a8544c4c2b7475fa936aacad789d1f5: Deleting consensus metadata
I20260928 15:55:30.506868 8044 catalog_manager.cc:4841] TS 3a8544c4c2b7475fa936aacad789d1f5 (127.22.157.195:40729): tablet fdfdfb65f70a4aeaaea0a3fee85fef64 (table test-table [id=f6cb7c52466f4ee88723537ae263e6f5]) successfully deleted
I20260928 15:55:30.507972 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.477148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.22.157.193:42205)
0928 15:55:30.477335 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.477341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.486672 (+ 9331us) server_negotiation.cc:407] Connection header received
0928 15:55:30.487433 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.487440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.487550 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.487637 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:30.490425 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.492309 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.496145 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.496611 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.496737 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.501213 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:30.501236 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:30.501254 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:30.501290 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:30.504618 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.505475 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.505485 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.505495 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.505681 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.506101 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.506107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.506110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.506526 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:30.507087 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.507433 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.507629 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 15:55:30.508798 8242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:30.619729 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 8119
I20260928 15:55:30.636348 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 8253
Sep 28 15:55:30 dist-test-slave-1vw0 krb5kdc[8013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:55:30.652017 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.637129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.22.157.194:38405)
0928 15:55:30.637330 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.637336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.637437 (+ 101us) server_negotiation.cc:407] Connection header received
0928 15:55:30.637702 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.637707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.637784 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.637888 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:30.639243 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.640056 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.641524 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.641933 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.642078 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.646363 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:30.646378 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:30.646393 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:30.646434 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:30.649440 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.650147 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.650156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.650158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.650293 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.650686 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.650690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.650692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.651070 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:30.651303 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.651576 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.651747 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:55:30.652812 8376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:30.745276 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 8253
I20260928 15:55:30.763108 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 8387
Sep 28 15:55:30 dist-test-slave-1vw0 krb5kdc[8013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:55:30.779516 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:30.764082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.22.157.195:40729)
0928 15:55:30.764303 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:30.764311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:30.764338 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:55:30.764563 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:30.764568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:30.764655 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:30.764781 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:30.766140 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.766902 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.768230 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:30.768664 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:30.768841 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:30.773694 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:30.773711 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:30.773727 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:30.773769 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:30.776503 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.777281 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.777289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.777291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.777437 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:30.777855 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:30.777864 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:30.777867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:30.778237 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:30.778545 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:30.778892 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 15:55:30.779135 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 15:55:30.780153 8510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:30.871764 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 8387
I20260928 15:55:30.890658 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 8026
I20260928 15:55:30.891886 8088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:31.012087 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 8026
2026-09-28T15:55:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4555 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T15:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:31Z Disabled control of system clock
I20260928 15:55:31.103288 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:46109
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:55:31.448853 8569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:31.449180 8569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:31.449357 8569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:31.457064 8569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:31.457192 8569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:31.457231 8569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:31.457335 8569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:31.457418 8569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:31.467206 8569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:46109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:31.468645 8569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:31.470677 8569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:31.483647 8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:31.483598 8577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:31.483556 8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:31.484701 8569 server_base.cc:1034] running on GCE node
I20260928 15:55:31.485706 8569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:31.487259 8569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:31.488469 8569 hybrid_clock.cc:648] HybridClock initialized: now 1790610931488396 us; error 62 us; skew 500 ppm
I20260928 15:55:31.488901 8569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:31.497596 8569 webserver.cc:460] Webserver started at http://127.22.157.254:42409/ using document root <none> and password file <none>
I20260928 15:55:31.498394 8569 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:31.498539 8569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:31.498895 8569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:31.502219 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "e8826374b5ea4356b153835c0e7f0674"
format_stamp: "Formatted at 2026-09-28 15:55:31 on dist-test-slave-1vw0"
server_key: "41ea0af68247b55e19764ced7a421d14"
server_key_iv: "58bf1b7c5689300d5a7627e1f266efda"
server_key_version: "encryptionkey@0"
I20260928 15:55:31.503083 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "e8826374b5ea4356b153835c0e7f0674"
format_stamp: "Formatted at 2026-09-28 15:55:31 on dist-test-slave-1vw0"
server_key: "41ea0af68247b55e19764ced7a421d14"
server_key_iv: "58bf1b7c5689300d5a7627e1f266efda"
server_key_version: "encryptionkey@0"
I20260928 15:55:31.508916 8569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:55:31.512856 8583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:31.514621 8569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:55:31.514859 8569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "e8826374b5ea4356b153835c0e7f0674"
format_stamp: "Formatted at 2026-09-28 15:55:31 on dist-test-slave-1vw0"
server_key: "41ea0af68247b55e19764ced7a421d14"
server_key_iv: "58bf1b7c5689300d5a7627e1f266efda"
server_key_version: "encryptionkey@0"
I20260928 15:55:31.515110 8569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:31.526160 8569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:31.526996 8569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:31.527297 8569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:31.546811 8569 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:46109
I20260928 15:55:31.546831 8634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:46109 every 8 connection(s)
I20260928 15:55:31.548247 8569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:31.552313 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 8569
I20260928 15:55:31.552503 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:31.552532 8635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:31.553200 23159 external_mini_cluster.cc:1428] Setting key 6bc020dca86d9f74335c66c75068373e
I20260928 15:55:31.566329 8635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674: Bootstrap starting.
I20260928 15:55:31.567559 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:31.555323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.22.157.254:46109)
0928 15:55:31.556518 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:31.556531 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:31.556575 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:55:31.558190 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:31.558212 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:31.558723 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:31.559215 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:31.559830 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:31.561260 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:31.562719 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:31.563455 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:31.563675 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:31.565094 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:31.565103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:31.565118 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:31.565694 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:31.565780 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:31.565802 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:31.565934 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:31.566196 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 15:55:31.566588 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":470,"threads_started":1}
I20260928 15:55:31.569773 8635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:31.570994 8635 log.cc:824] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:31.574422 8635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674: No bootstrap required, opened a new log
I20260928 15:55:31.582512 8635 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826374b5ea4356b153835c0e7f0674" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46109 } }
I20260928 15:55:31.583009 8635 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:31.583133 8635 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8826374b5ea4356b153835c0e7f0674, State: Initialized, Role: FOLLOWER
I20260928 15:55:31.583727 8635 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826374b5ea4356b153835c0e7f0674" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46109 } }
I20260928 15:55:31.584028 8635 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:31.584167 8635 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:31.584353 8635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:31.586714 8635 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826374b5ea4356b153835c0e7f0674" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46109 } }
I20260928 15:55:31.587286 8635 leader_election.cc:302] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8826374b5ea4356b153835c0e7f0674; no voters:
I20260928 15:55:31.587918 8635 leader_election.cc:288] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:31.588119 8640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:31.588913 8640 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [term 1 LEADER]: Becoming Leader. State: Replica: e8826374b5ea4356b153835c0e7f0674, State: Running, Role: LEADER
I20260928 15:55:31.589490 8640 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826374b5ea4356b153835c0e7f0674" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46109 } }
I20260928 15:55:31.590130 8635 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:31.592790 8641 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8826374b5ea4356b153835c0e7f0674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826374b5ea4356b153835c0e7f0674" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46109 } } }
I20260928 15:55:31.593318 8641 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:31.592967 8642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8826374b5ea4356b153835c0e7f0674. Latest consensus state: current_term: 1 leader_uuid: "e8826374b5ea4356b153835c0e7f0674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8826374b5ea4356b153835c0e7f0674" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 46109 } } }
I20260928 15:55:31.593497 8642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:31.594671 8646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:31.600854 8646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:31.610530 8646 catalog_manager.cc:1347] Generated new cluster ID: 425ac475fdd647d0bdecde0bb1dd5d6f
I20260928 15:55:31.610720 8646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:31.640769 8646 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:31.642165 8646 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:31.656924 8646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674: Generated new TSK 0
I20260928 15:55:31.657996 8646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:55:31.663763 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46109
--builtin_ntp_servers=127.22.157.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:55:32.035244 8659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:32.035606 8659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:32.035737 8659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:32.043185 8659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:32.043299 8659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:32.043407 8659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:55:32.052700 8659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:32.054013 8659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:32.055904 8659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:32.070541 8665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:32.070698 8667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:32.070541 8664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:32.071530 8659 server_base.cc:1034] running on GCE node
I20260928 15:55:32.072252 8659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:32.073493 8659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:32.074757 8659 hybrid_clock.cc:648] HybridClock initialized: now 1790610932074683 us; error 67 us; skew 500 ppm
I20260928 15:55:32.075145 8659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:32.078017 8659 webserver.cc:460] Webserver started at http://127.22.157.193:44869/ using document root <none> and password file <none>
I20260928 15:55:32.078810 8659 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:32.078954 8659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:32.079293 8659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:32.082536 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "af894d30b60b458c945fc06f2445ec55"
format_stamp: "Formatted at 2026-09-28 15:55:32 on dist-test-slave-1vw0"
server_key: "eda5263a8b8445ef54f20f06777d81ec"
server_key_iv: "0aadbce8933b082d7e77ed883d323fe1"
server_key_version: "encryptionkey@0"
I20260928 15:55:32.083422 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "af894d30b60b458c945fc06f2445ec55"
format_stamp: "Formatted at 2026-09-28 15:55:32 on dist-test-slave-1vw0"
server_key: "eda5263a8b8445ef54f20f06777d81ec"
server_key_iv: "0aadbce8933b082d7e77ed883d323fe1"
server_key_version: "encryptionkey@0"
I20260928 15:55:32.089375 8659 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 15:55:32.093626 8673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:32.095204 8659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 15:55:32.095441 8659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "af894d30b60b458c945fc06f2445ec55"
format_stamp: "Formatted at 2026-09-28 15:55:32 on dist-test-slave-1vw0"
server_key: "eda5263a8b8445ef54f20f06777d81ec"
server_key_iv: "0aadbce8933b082d7e77ed883d323fe1"
server_key_version: "encryptionkey@0"
I20260928 15:55:32.095736 8659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:32.125674 8659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:32.126591 8659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:32.126905 8659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:32.128018 8659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:32.129549 8659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:32.129679 8659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:32.129819 8659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:32.129910 8659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:32.163903 8659 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:35453
I20260928 15:55:32.163995 8785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:35453 every 8 connection(s)
I20260928 15:55:32.165411 8659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:55:32.168406 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 8659
I20260928 15:55:32.168605 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:55:32.169235 23159 external_mini_cluster.cc:1428] Setting key c78f0c10a1ae6fc57ed8252c5d57abc6
I20260928 15:55:32.172191 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46109
--builtin_ntp_servers=127.22.157.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:55:32.180522 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:32.168579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:36345 (local address 127.22.157.254:46109)
0928 15:55:32.169109 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:32.169115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:32.170600 (+ 1485us) server_negotiation.cc:407] Connection header received
0928 15:55:32.172539 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:32.172548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:32.172655 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:32.172829 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:32.173986 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:32.175315 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:32.177009 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:32.177817 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:32.177978 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:32.179410 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:32.179420 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:32.179429 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:32.179459 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:32.179527 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:32.179548 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:32.179653 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:32.179891 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 15:55:32.180015 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":257,"threads_started":1}
I20260928 15:55:32.181356 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:32.169300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46109 (local address 127.22.157.193:36345)
0928 15:55:32.170417 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:32.170463 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:32.172148 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:32.173018 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:32.173030 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:32.173127 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:32.173798 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:32.173821 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:32.175479 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:32.175483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:32.176863 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:32.176875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:32.178062 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:32.178068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:32.178283 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:32.179057 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:55:32.179059 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:32.179107 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:55:32.179192 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:32.179656 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:32.179666 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:32.179671 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 15:55:32.179840 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 15:55:32.180169 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":345,"threads_started":1}
I20260928 15:55:32.183244 8786 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46109
I20260928 15:55:32.183634 8786 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:32.184432 8786 heartbeater.cc:511] Master 127.22.157.254:46109 requested a full tablet report, sending...
I20260928 15:55:32.186983 8600 ts_manager.cc:194] Registered new tserver with Master: af894d30b60b458c945fc06f2445ec55 (127.22.157.193:35453)
I20260928 15:55:32.189219 8600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.193:36345
W20260928 15:55:32.511857 8790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:32.512207 8790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:32.512305 8790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:32.520553 8790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:32.520808 8790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:32.520922 8790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:55:32.530849 8790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:32.532353 8790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:32.534185 8790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:32.548391 8796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:32.548431 8798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:32.549149 8795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:32.554804 8790 server_base.cc:1034] running on GCE node
I20260928 15:55:32.555616 8790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:32.556823 8790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:32.558084 8790 hybrid_clock.cc:648] HybridClock initialized: now 1790610932558032 us; error 47 us; skew 500 ppm
I20260928 15:55:32.558463 8790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:32.561543 8790 webserver.cc:460] Webserver started at http://127.22.157.194:40007/ using document root <none> and password file <none>
I20260928 15:55:32.562388 8790 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:32.562530 8790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:32.562892 8790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:32.566206 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "53b0ffa6588545429c7c140797d629dd"
format_stamp: "Formatted at 2026-09-28 15:55:32 on dist-test-slave-1vw0"
server_key: "311d59544056d5d6399a4189f2142fb9"
server_key_iv: "d808aa7357a2bb636cf0bf20e6977487"
server_key_version: "encryptionkey@0"
I20260928 15:55:32.567116 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "53b0ffa6588545429c7c140797d629dd"
format_stamp: "Formatted at 2026-09-28 15:55:32 on dist-test-slave-1vw0"
server_key: "311d59544056d5d6399a4189f2142fb9"
server_key_iv: "d808aa7357a2bb636cf0bf20e6977487"
server_key_version: "encryptionkey@0"
I20260928 15:55:32.572985 8790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 15:55:32.577168 8804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:32.578670 8790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:55:32.578860 8790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "53b0ffa6588545429c7c140797d629dd"
format_stamp: "Formatted at 2026-09-28 15:55:32 on dist-test-slave-1vw0"
server_key: "311d59544056d5d6399a4189f2142fb9"
server_key_iv: "d808aa7357a2bb636cf0bf20e6977487"
server_key_version: "encryptionkey@0"
I20260928 15:55:32.579123 8790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:32.597729 8790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:32.598628 8790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:32.598953 8790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:32.600081 8790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:32.601817 8790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:32.601912 8790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:32.602059 8790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:32.602157 8790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:32.636785 8790 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:37855
I20260928 15:55:32.636850 8916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:37855 every 8 connection(s)
I20260928 15:55:32.638270 8790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:55:32.644650 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 8790
I20260928 15:55:32.645008 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:55:32.645535 23159 external_mini_cluster.cc:1428] Setting key 1b37737e6a7cfffc13b06ba3d83e0593
I20260928 15:55:32.648418 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46109
--builtin_ntp_servers=127.22.157.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:55:32.652751 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:32.640996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:36177 (local address 127.22.157.254:46109)
0928 15:55:32.641207 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:32.641211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:32.642581 (+ 1370us) server_negotiation.cc:407] Connection header received
0928 15:55:32.645021 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:32.645027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:32.645156 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:32.645297 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:32.646717 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:32.647720 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:32.649649 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:32.650112 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:32.650258 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:32.651788 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:32.651793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:32.651797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:32.651816 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:32.651850 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:32.651862 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:32.651966 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:32.652242 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 15:55:32.652352 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":75}
I20260928 15:55:32.653909 8919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:32.641435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46109 (local address 127.22.157.194:36177)
0928 15:55:32.642359 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:32.642402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:32.644576 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:32.645573 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:32.645582 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:32.645730 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:32.646529 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:32.646562 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:32.647883 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:32.647889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:32.649493 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:32.649507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:32.650367 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:32.650375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:32.650642 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:32.651418 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:55:32.651420 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:32.651465 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:55:32.651564 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:32.651964 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:32.651976 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:32.651986 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 15:55:32.652231 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 15:55:32.652697 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":266,"threads_started":1}
I20260928 15:55:32.655579 8917 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46109
I20260928 15:55:32.655917 8917 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:32.656791 8917 heartbeater.cc:511] Master 127.22.157.254:46109 requested a full tablet report, sending...
I20260928 15:55:32.658344 8600 ts_manager.cc:194] Registered new tserver with Master: 53b0ffa6588545429c7c140797d629dd (127.22.157.194:37855)
I20260928 15:55:32.659219 8600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.194:36177
W20260928 15:55:32.992527 8920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:32.992934 8920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:32.993108 8920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:33.000658 8920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:33.000829 8920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:33.001032 8920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:55:33.010543 8920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:33.011891 8920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:33.013821 8920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:33.027055 8928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:33.027115 8925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:33.027031 8926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:33.028704 8920 server_base.cc:1034] running on GCE node
I20260928 15:55:33.029572 8920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:33.030594 8920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:33.031844 8920 hybrid_clock.cc:648] HybridClock initialized: now 1790610933031784 us; error 58 us; skew 500 ppm
I20260928 15:55:33.032178 8920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:33.034714 8920 webserver.cc:460] Webserver started at http://127.22.157.195:45685/ using document root <none> and password file <none>
I20260928 15:55:33.035454 8920 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:33.035588 8920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:33.035908 8920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:33.039047 8920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef"
format_stamp: "Formatted at 2026-09-28 15:55:33 on dist-test-slave-1vw0"
server_key: "1ff8ce3f8f1197de9ad0efb939ee0477"
server_key_iv: "0658a0ed6071a45995cfd54b7162f83a"
server_key_version: "encryptionkey@0"
I20260928 15:55:33.039842 8920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef"
format_stamp: "Formatted at 2026-09-28 15:55:33 on dist-test-slave-1vw0"
server_key: "1ff8ce3f8f1197de9ad0efb939ee0477"
server_key_iv: "0658a0ed6071a45995cfd54b7162f83a"
server_key_version: "encryptionkey@0"
I20260928 15:55:33.045181 8920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 15:55:33.048812 8934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:33.050172 8920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:55:33.050349 8920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef"
format_stamp: "Formatted at 2026-09-28 15:55:33 on dist-test-slave-1vw0"
server_key: "1ff8ce3f8f1197de9ad0efb939ee0477"
server_key_iv: "0658a0ed6071a45995cfd54b7162f83a"
server_key_version: "encryptionkey@0"
I20260928 15:55:33.050621 8920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:33.061986 8920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:33.062775 8920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:33.063089 8920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:33.064033 8920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:33.065629 8920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:33.065750 8920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:33.065881 8920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:33.065965 8920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:33.100324 8920 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:43731
I20260928 15:55:33.100432 9046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:43731 every 8 connection(s)
I20260928 15:55:33.101748 8920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:55:33.107743 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 8920
I20260928 15:55:33.107944 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:55:33.108415 23159 external_mini_cluster.cc:1428] Setting key 35d2e415a53bbdf4b0fac59313c42e5d
I20260928 15:55:33.115480 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.104542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:57355 (local address 127.22.157.254:46109)
0928 15:55:33.104817 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.104822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.106434 (+ 1612us) server_negotiation.cc:407] Connection header received
0928 15:55:33.108421 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.108427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.108530 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.108659 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:33.110002 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.110910 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.112614 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.113088 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.113224 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.114619 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:33.114624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:33.114627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:33.114644 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:33.114678 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:33.114690 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:33.114789 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.115025 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.115132 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 15:55:33.116631 9049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.105269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:46109 (local address 127.22.157.195:57355)
0928 15:55:33.106257 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.106295 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:33.108131 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:33.108968 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:33.108981 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:33.109131 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:33.109814 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.109849 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.111089 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.111097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.112457 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.112469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.113326 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.113334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.113583 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.114265 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:55:33.114267 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:33.114314 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:55:33.114395 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:33.114786 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:33.114798 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:33.114809 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:55:33.115024 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 15:55:33.115443 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":299,"threads_started":1}
I20260928 15:55:33.118214 9047 heartbeater.cc:347] Connected to a master server at 127.22.157.254:46109
I20260928 15:55:33.118566 9047 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:33.119449 9047 heartbeater.cc:511] Master 127.22.157.254:46109 requested a full tablet report, sending...
I20260928 15:55:33.121021 8600 ts_manager.cc:194] Registered new tserver with Master: a0c95cf5a2b547ceaf2d9bd11dedddef (127.22.157.195:43731)
I20260928 15:55:33.121928 8600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.195:57355
I20260928 15:55:33.124332 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:55:33.134616 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.128906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.22.157.254:46109)
0928 15:55:33.129091 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.129096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.129379 (+ 283us) server_negotiation.cc:407] Connection header received
0928 15:55:33.129526 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.129529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.129630 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.129718 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:33.130215 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.130930 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.132310 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.132763 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.132854 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.133984 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:33.133988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:33.133991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:33.134009 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:33.134043 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:33.134059 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:33.134129 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.134266 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.134340 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:55:33.138939 8600 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:55:33.145931 8600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:55:33.182595 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.159054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:43731 (local address 127.0.0.1:54226)
0928 15:55:33.172047 (+ 12993us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.172088 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:33.172297 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:33.174661 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:33.174671 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:33.174706 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:33.175124 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.175143 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.176669 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.176677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.179477 (+ 2800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.179494 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.181413 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.181421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.181679 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.181741 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 15:55:33.181878 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 15:55:33.182137 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12765,"mutex_wait_us":1,"thread_start_us":316,"threads_started":1}
I20260928 15:55:33.182750 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.159673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.22.157.195:43731)
0928 15:55:33.173076 (+ 13403us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.173090 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.173141 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:55:33.173288 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.173300 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.173665 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.173920 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:33.175393 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.176498 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.179803 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.181148 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.181396 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.181671 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.182176 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.182355 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13146,"thread_start_us":12940,"threads_started":1}
I20260928 15:55:33.185782 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.161006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:37855 (local address 127.0.0.1:39062)
0928 15:55:33.174091 (+ 13085us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.174119 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:33.174265 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:33.177824 (+ 3559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:33.177830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:33.177861 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:33.178227 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.178244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.180259 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.180264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.182689 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.182704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.184905 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.184912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.185092 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.185128 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:55:33.185236 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 15:55:33.185380 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12934,"mutex_wait_us":92,"thread_start_us":13174,"threads_started":1}
I20260928 15:55:33.185896 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.161215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.22.157.194:37855)
0928 15:55:33.177145 (+ 15930us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.177153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.177184 (+ 31us) server_negotiation.cc:407] Connection header received
0928 15:55:33.177288 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.177296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.177590 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.178450 (+ 860us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:33.178680 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.180110 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.183179 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.184682 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.184869 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.185110 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.185288 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.185462 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15746,"thread_start_us":15632,"threads_started":1}
I20260928 15:55:33.187508 8982 tablet_service.cc:1467] Processing CreateTablet for tablet ab7b4610bd25447f9c7a20c0e61a4297 (DEFAULT_TABLE table=test-table [id=dd4a00827e7b471e90d3e65a356f9c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:33.188369 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.158907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35453 (local address 127.0.0.1:51574)
0928 15:55:33.166582 (+ 7675us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.166646 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:33.166843 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:33.180966 (+ 14123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:33.180971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:33.181003 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:33.181651 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.181666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.183919 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.183924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.186082 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.186096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.187591 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.187598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.187769 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.187811 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 15:55:33.187892 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:55:33.188014 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7342,"mutex_wait_us":25,"thread_start_us":15448,"threads_started":1}
I20260928 15:55:33.189333 8982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7b4610bd25447f9c7a20c0e61a4297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:33.189352 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.159127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.22.157.193:35453)
0928 15:55:33.180225 (+ 21098us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.180235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.180264 (+ 29us) server_negotiation.cc:407] Connection header received
0928 15:55:33.180376 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.180384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.180661 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.180960 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:33.181860 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.183754 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.186305 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.187366 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.187747 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.187970 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.188838 (+ 868us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.189026 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20875,"thread_start_us":20758,"threads_started":1}
I20260928 15:55:33.190259 8852 tablet_service.cc:1467] Processing CreateTablet for tablet ab7b4610bd25447f9c7a20c0e61a4297 (DEFAULT_TABLE table=test-table [id=dd4a00827e7b471e90d3e65a356f9c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:33.191769 8852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7b4610bd25447f9c7a20c0e61a4297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:33.192750 8786 heartbeater.cc:503] Master 127.22.157.254:46109 was elected leader, sending a full tablet report...
I20260928 15:55:33.193172 8721 tablet_service.cc:1467] Processing CreateTablet for tablet ab7b4610bd25447f9c7a20c0e61a4297 (DEFAULT_TABLE table=test-table [id=dd4a00827e7b471e90d3e65a356f9c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:33.195633 8721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7b4610bd25447f9c7a20c0e61a4297. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:33.203649 9065 tablet_bootstrap.cc:492] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: Bootstrap starting.
I20260928 15:55:33.206872 9065 tablet_bootstrap.cc:654] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:33.207784 9066 tablet_bootstrap.cc:492] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: Bootstrap starting.
I20260928 15:55:33.208235 9065 log.cc:824] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:33.211488 9066 tablet_bootstrap.cc:654] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:33.211683 9065 tablet_bootstrap.cc:492] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: No bootstrap required, opened a new log
I20260928 15:55:33.211999 9065 ts_tablet_manager.cc:1397] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 15:55:33.212947 9066 log.cc:824] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:33.214463 9068 tablet_bootstrap.cc:492] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55: Bootstrap starting.
I20260928 15:55:33.216059 9066 tablet_bootstrap.cc:492] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: No bootstrap required, opened a new log
I20260928 15:55:33.216347 9066 ts_tablet_manager.cc:1397] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 15:55:33.218271 9068 tablet_bootstrap.cc:654] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:33.219590 9068 log.cc:824] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:33.222993 9065 raft_consensus.cc:348] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } }
I20260928 15:55:33.223363 9068 tablet_bootstrap.cc:492] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55: No bootstrap required, opened a new log
I20260928 15:55:33.223500 9065 raft_consensus.cc:374] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:33.223659 9068 ts_tablet_manager.cc:1397] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 15:55:33.223670 9065 raft_consensus.cc:729] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c95cf5a2b547ceaf2d9bd11dedddef, State: Initialized, Role: FOLLOWER
I20260928 15:55:33.224345 9065 consensus_queue.cc:260] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } }
I20260928 15:55:33.225260 9066 raft_consensus.cc:348] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } }
I20260928 15:55:33.225776 9066 raft_consensus.cc:374] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:33.225766 9047 heartbeater.cc:503] Master 127.22.157.254:46109 was elected leader, sending a full tablet report...
I20260928 15:55:33.225930 9066 raft_consensus.cc:729] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b0ffa6588545429c7c140797d629dd, State: Initialized, Role: FOLLOWER
I20260928 15:55:33.226338 9065 ts_tablet_manager.cc:1428] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260928 15:55:33.226599 9066 consensus_queue.cc:260] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } }
I20260928 15:55:33.227692 8917 heartbeater.cc:503] Master 127.22.157.254:46109 was elected leader, sending a full tablet report...
I20260928 15:55:33.228398 9066 ts_tablet_manager.cc:1428] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260928 15:55:33.234593 9068 raft_consensus.cc:348] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } }
I20260928 15:55:33.235177 9068 raft_consensus.cc:374] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:33.235313 9068 raft_consensus.cc:729] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af894d30b60b458c945fc06f2445ec55, State: Initialized, Role: FOLLOWER
I20260928 15:55:33.235970 9068 consensus_queue.cc:260] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } }
I20260928 15:55:33.237743 9068 ts_tablet_manager.cc:1428] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
W20260928 15:55:33.353564 9048 tablet.cc:2369] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:55:33.355069 9073 raft_consensus.cc:482] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:55:33.355479 9073 raft_consensus.cc:504] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } }
I20260928 15:55:33.357576 9073 leader_election.cc:288] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0c95cf5a2b547ceaf2d9bd11dedddef (127.22.157.195:43731), 53b0ffa6588545429c7c140797d629dd (127.22.157.194:37855)
I20260928 15:55:33.365236 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.357834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:53873 (local address 127.22.157.195:43731)
0928 15:55:33.358073 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.358080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.358796 (+ 716us) server_negotiation.cc:407] Connection header received
0928 15:55:33.359039 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.359047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.359145 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.359253 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:33.359915 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.360933 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.363134 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.364232 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.364421 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.364503 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.364871 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.364993 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:55:33.365250 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.358303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:40599 (local address 127.22.157.194:37855)
0928 15:55:33.358534 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.358543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.359161 (+ 618us) server_negotiation.cc:407] Connection header received
0928 15:55:33.359351 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.359357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.359445 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.359601 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:33.360240 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.361071 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.363248 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.364207 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.364351 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.364429 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.364818 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.364961 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 15:55:33.365356 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.358069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:37855 (local address 127.22.157.193:40599)
0928 15:55:33.359027 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.359047 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:33.359191 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:33.359579 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:33.359583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:33.359605 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:33.360065 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.360080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.361242 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.361246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.363018 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.363052 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.364409 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.364415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.364611 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.364666 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 15:55:33.364787 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 15:55:33.364946 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":52,"thread_start_us":542,"threads_started":1}
I20260928 15:55:33.365408 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.357747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:43731 (local address 127.22.157.193:53873)
0928 15:55:33.358652 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.358674 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:33.358862 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:33.359302 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:33.359307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:33.359338 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:33.359744 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.359758 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.361091 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.361096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.362887 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:33.362900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.364462 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.364469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:33.364653 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.364680 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 15:55:33.364829 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 15:55:33.364960 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":75,"thread_start_us":510,"threads_started":1}
I20260928 15:55:33.366442 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7b4610bd25447f9c7a20c0e61a4297" candidate_uuid: "af894d30b60b458c945fc06f2445ec55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b0ffa6588545429c7c140797d629dd" is_pre_election: true
I20260928 15:55:33.366668 9002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7b4610bd25447f9c7a20c0e61a4297" candidate_uuid: "af894d30b60b458c945fc06f2445ec55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" is_pre_election: true
I20260928 15:55:33.367093 8872 raft_consensus.cc:2454] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af894d30b60b458c945fc06f2445ec55 in term 0.
I20260928 15:55:33.367195 9002 raft_consensus.cc:2454] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af894d30b60b458c945fc06f2445ec55 in term 0.
I20260928 15:55:33.367837 8677 leader_election.cc:302] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c95cf5a2b547ceaf2d9bd11dedddef, af894d30b60b458c945fc06f2445ec55; no voters:
I20260928 15:55:33.368251 9073 raft_consensus.cc:2781] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:55:33.368402 9073 raft_consensus.cc:482] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:55:33.368507 9073 raft_consensus.cc:3037] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:33.370735 9073 raft_consensus.cc:504] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } }
I20260928 15:55:33.371373 9073 leader_election.cc:288] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [CANDIDATE]: Term 1 election: Requested vote from peers a0c95cf5a2b547ceaf2d9bd11dedddef (127.22.157.195:43731), 53b0ffa6588545429c7c140797d629dd (127.22.157.194:37855)
I20260928 15:55:33.371842 9002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7b4610bd25447f9c7a20c0e61a4297" candidate_uuid: "af894d30b60b458c945fc06f2445ec55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef"
I20260928 15:55:33.372061 9002 raft_consensus.cc:3037] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:33.372097 8872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7b4610bd25447f9c7a20c0e61a4297" candidate_uuid: "af894d30b60b458c945fc06f2445ec55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b0ffa6588545429c7c140797d629dd"
I20260928 15:55:33.372367 8872 raft_consensus.cc:3037] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:33.374373 9002 raft_consensus.cc:2454] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af894d30b60b458c945fc06f2445ec55 in term 1.
I20260928 15:55:33.374826 8677 leader_election.cc:302] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0c95cf5a2b547ceaf2d9bd11dedddef, af894d30b60b458c945fc06f2445ec55; no voters:
I20260928 15:55:33.375098 8872 raft_consensus.cc:2454] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af894d30b60b458c945fc06f2445ec55 in term 1.
I20260928 15:55:33.375129 9073 raft_consensus.cc:2781] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:33.375751 9073 raft_consensus.cc:686] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 1 LEADER]: Becoming Leader. State: Replica: af894d30b60b458c945fc06f2445ec55, State: Running, Role: LEADER
I20260928 15:55:33.376268 9073 consensus_queue.cc:237] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } }
I20260928 15:55:33.381395 8599 catalog_manager.cc:5495] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 reported cstate change: term changed from 0 to 1, leader changed from <none> to af894d30b60b458c945fc06f2445ec55 (127.22.157.193). New cstate: current_term: 1 leader_uuid: "af894d30b60b458c945fc06f2445ec55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af894d30b60b458c945fc06f2445ec55" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 35453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 } health_report { overall_health: UNKNOWN } } }
W20260928 15:55:33.390462 8918 tablet.cc:2369] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:55:33.418165 8787 tablet.cc:2369] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:55:33.423593 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.417574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.22.157.193:35453)
0928 15:55:33.417829 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.417836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.417858 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:55:33.417977 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.417982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.418181 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.418355 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:55:33.418889 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.419702 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.421210 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.421624 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.421718 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.422316 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:55:33.422929 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:55:33.423054 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.423174 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.423245 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:55:33.432364 8872 raft_consensus.cc:1261] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [term 1 FOLLOWER]: Refusing update from remote peer af894d30b60b458c945fc06f2445ec55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:33.432367 9002 raft_consensus.cc:1261] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [term 1 FOLLOWER]: Refusing update from remote peer af894d30b60b458c945fc06f2445ec55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:33.433475 9076 consensus_queue.cc:1035] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b0ffa6588545429c7c140797d629dd" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 37855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:33.433838 9073 consensus_queue.cc:1035] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0c95cf5a2b547ceaf2d9bd11dedddef" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:33.469658 8599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:33.469951 8599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:33.473618 8599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8826374b5ea4356b153835c0e7f0674: Sending DeleteTablet for 3 replicas of tablet ab7b4610bd25447f9c7a20c0e61a4297
I20260928 15:55:33.475008 8982 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7b4610bd25447f9c7a20c0e61a4297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:33 UTC) from {username='slave'} at 127.0.0.1:54226
I20260928 15:55:33.475028 8721 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7b4610bd25447f9c7a20c0e61a4297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:33 UTC) from {username='slave'} at 127.0.0.1:51574
I20260928 15:55:33.475270 8852 tablet_service.cc:1514] Processing DeleteTablet for tablet ab7b4610bd25447f9c7a20c0e61a4297 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:33 UTC) from {username='slave'} at 127.0.0.1:39062
I20260928 15:55:33.477737 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 8659
I20260928 15:55:33.487510 9090 tablet_replica.cc:330] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: stopping tablet replica
I20260928 15:55:33.489375 9090 raft_consensus.cc:2229] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:33.490785 9090 raft_consensus.cc:2258] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:33.494752 9090 ts_tablet_manager.cc:1905] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:33.495908 9091 tablet_replica.cc:330] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55: stopping tablet replica
I20260928 15:55:33.497225 9091 raft_consensus.cc:2229] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:55:33.498476 9091 raft_consensus.cc:2258] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:33.501327 9092 tablet_replica.cc:330] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: stopping tablet replica
I20260928 15:55:33.502112 9091 ts_tablet_manager.cc:1905] T ab7b4610bd25447f9c7a20c0e61a4297 P af894d30b60b458c945fc06f2445ec55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:33.503542 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.479833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.22.157.193:35453)
0928 15:55:33.480506 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.480517 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.493255 (+ 12738us) server_negotiation.cc:407] Connection header received
0928 15:55:33.493832 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.493838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.493958 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.494046 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:33.494814 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.496235 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.498361 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.499048 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.499152 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.501622 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:33.501634 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:33.501661 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:33.502469 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:33.502531 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:33.502557 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:33.502647 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.503012 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.503136 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492}
I20260928 15:55:33.504716 9090 ts_tablet_manager.cc:1918] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:33.505000 9090 log.cc:1196] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/ab7b4610bd25447f9c7a20c0e61a4297
I20260928 15:55:33.505003 8781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:33.505616 9090 ts_tablet_manager.cc:1939] T ab7b4610bd25447f9c7a20c0e61a4297 P a0c95cf5a2b547ceaf2d9bd11dedddef: Deleting consensus metadata
I20260928 15:55:33.505505 9092 raft_consensus.cc:2229] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:33.506814 9092 raft_consensus.cc:2258] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:33.508623 8587 catalog_manager.cc:4841] TS a0c95cf5a2b547ceaf2d9bd11dedddef (127.22.157.195:43731): tablet ab7b4610bd25447f9c7a20c0e61a4297 (table test-table [id=dd4a00827e7b471e90d3e65a356f9c6c]) successfully deleted
I20260928 15:55:33.509624 9092 ts_tablet_manager.cc:1905] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:33.514846 9092 ts_tablet_manager.cc:1918] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:33.514992 9092 log.cc:1196] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/ab7b4610bd25447f9c7a20c0e61a4297
I20260928 15:55:33.515470 9092 ts_tablet_manager.cc:1939] T ab7b4610bd25447f9c7a20c0e61a4297 P 53b0ffa6588545429c7c140797d629dd: Deleting consensus metadata
I20260928 15:55:33.517192 8585 catalog_manager.cc:4841] TS 53b0ffa6588545429c7c140797d629dd (127.22.157.194:37855): tablet ab7b4610bd25447f9c7a20c0e61a4297 (table test-table [id=dd4a00827e7b471e90d3e65a356f9c6c]) successfully deleted
I20260928 15:55:33.606318 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 8659
I20260928 15:55:33.626479 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 8790
W20260928 15:55:33.626420 8584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.157.193:35453 (error 108)
I20260928 15:55:33.628506 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.627932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35453 (local address 127.0.0.1:51610)
0928 15:55:33.628169 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.628310 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260928 15:55:33.628826 8584 catalog_manager.cc:4584] TS af894d30b60b458c945fc06f2445ec55 (127.22.157.193:35453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab7b4610bd25447f9c7a20c0e61a4297: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35453: connect: Connection refused (error 111)
I20260928 15:55:33.633929 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.627455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.22.157.194:37855)
0928 15:55:33.627663 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.627692 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.627715 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:55:33.627882 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.627888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.628013 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.628135 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:33.628660 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.629533 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.630821 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.631256 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.631362 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.632441 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:33.632449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:33.632510 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:33.633283 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:33.633337 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:33.633358 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:33.633444 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.633570 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.633665 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 15:55:33.634696 8912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:33.682538 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.681965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35453 (local address 127.0.0.1:51624)
0928 15:55:33.682222 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.682328 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 15:55:33.683905 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.683476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35453 (local address 127.0.0.1:51636)
0928 15:55:33.683658 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.683753 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:55:33.724839 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 8790
I20260928 15:55:33.743955 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 8920
I20260928 15:55:33.744791 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.744235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35453 (local address 127.0.0.1:51644)
0928 15:55:33.744449 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.744564 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 15:55:33.745988 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.745558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35453 (local address 127.0.0.1:51646)
0928 15:55:33.745740 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.745844 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 15:55:33.751024 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.744804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.22.157.195:43731)
0928 15:55:33.745019 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:33.745025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:33.745047 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:55:33.745195 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:33.745208 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:33.745301 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:33.745432 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:33.745943 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.746744 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.748055 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:33.748440 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:33.748544 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:33.749614 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:33.749623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:33.749638 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:33.750218 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:33.750264 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:33.750284 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:33.750369 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:33.750584 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 15:55:33.750674 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:55:33.751765 9042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:33.826939 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.826411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35453 (local address 127.0.0.1:51662)
0928 15:55:33.826663 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.826767 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 15:55:33.828100 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:33.827672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:35453 (local address 127.0.0.1:51666)
0928 15:55:33.827882 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:33.827966 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:35453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 15:55:33.848197 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 8920
I20260928 15:55:33.865442 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 8569
I20260928 15:55:33.866562 8630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:33.980882 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 8569
2026-09-28T15:55:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2964 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:55:34 dist-test-slave-1vw0 krb5kdc[9101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:55:34 dist-test-slave-1vw0 krb5kdc[9101](info): set up 2 sockets
Sep 28 15:55:34 dist-test-slave-1vw0 krb5kdc[9101](info): commencing operation
krb5kdc: starting...
W20260928 15:55:35.117414 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:55:35 dist-test-slave-1vw0 krb5kdc[9101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:55:35.404549 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:36395
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:35.763157 9114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:35.763515 9114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:35.763684 9114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:35.771102 9114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:35.771271 9114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:35.771384 9114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:35.771485 9114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:35.771570 9114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:35.781997 9114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:36395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:35.783450 9114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:35.784864 9114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:35.798942 9119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:35.799021 9122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:35.798969 9120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:35.800295 9114 server_base.cc:1034] running on GCE node
I20260928 15:55:35.801204 9114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:35.802801 9114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:35.804018 9114 hybrid_clock.cc:648] HybridClock initialized: now 1790610935803971 us; error 35 us; skew 500 ppm
I20260928 15:55:35.804396 9114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:35 dist-test-slave-1vw0 krb5kdc[9101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610935, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:35.807636 9114 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:55:35.809301 9114 webserver.cc:460] Webserver started at http://127.22.157.254:44325/ using document root <none> and password file <none>
I20260928 15:55:35.810011 9114 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:35.810148 9114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:35.810494 9114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:35.813825 9114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "1a79977370da41839799f37406b2ad05"
format_stamp: "Formatted at 2026-09-28 15:55:35 on dist-test-slave-1vw0"
server_key: "cbac8552f915cb7114da18d4419e0f2f"
server_key_iv: "132423639504a7b498b4623204d77689"
server_key_version: "encryptionkey@0"
I20260928 15:55:35.814702 9114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "1a79977370da41839799f37406b2ad05"
format_stamp: "Formatted at 2026-09-28 15:55:35 on dist-test-slave-1vw0"
server_key: "cbac8552f915cb7114da18d4419e0f2f"
server_key_iv: "132423639504a7b498b4623204d77689"
server_key_version: "encryptionkey@0"
I20260928 15:55:35.820513 9114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:55:35.824709 9129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:35.826485 9114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:35.826699 9114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "1a79977370da41839799f37406b2ad05"
format_stamp: "Formatted at 2026-09-28 15:55:35 on dist-test-slave-1vw0"
server_key: "cbac8552f915cb7114da18d4419e0f2f"
server_key_iv: "132423639504a7b498b4623204d77689"
server_key_version: "encryptionkey@0"
I20260928 15:55:35.826931 9114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:35.843014 9114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:35.847537 9114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:35.847868 9114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:35.868163 9114 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:36395
I20260928 15:55:35.868207 9180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:36395 every 8 connection(s)
I20260928 15:55:35.869763 9114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:35.873790 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 9114
I20260928 15:55:35.873987 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:35.873965 9181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:35.874432 23159 external_mini_cluster.cc:1428] Setting key e186af78d33fe15b3ef032fe6bb42505
I20260928 15:55:35.890476 9181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05: Bootstrap starting.
Sep 28 15:55:35 dist-test-slave-1vw0 krb5kdc[9101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:35.894802 9181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:35.896121 9181 log.cc:824] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:35.899518 9181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05: No bootstrap required, opened a new log
I20260928 15:55:35.900848 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:35.876583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.22.157.254:36395)
0928 15:55:35.877595 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:35.877622 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:35.877691 (+ 69us) server_negotiation.cc:407] Connection header received
0928 15:55:35.879839 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:35.879873 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:35.880521 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:35.881083 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:35.882614 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:35.884714 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:35.886184 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:35.887458 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:35.887695 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:35.892634 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:35.892644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:35.892663 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:35.892710 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:35.896253 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:35.897179 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:35.897189 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:35.897198 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:35.897402 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:35.897814 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:35.897819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:35.897823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:35.898433 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:35.898793 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:35.899235 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 15:55:35.899734 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":369,"threads_started":1}
I20260928 15:55:35.908272 9181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a79977370da41839799f37406b2ad05" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36395 } }
I20260928 15:55:35.908865 9181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:35.909009 9181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a79977370da41839799f37406b2ad05, State: Initialized, Role: FOLLOWER
I20260928 15:55:35.909665 9181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a79977370da41839799f37406b2ad05" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36395 } }
I20260928 15:55:35.909945 9181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:35.910115 9181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:35.910301 9181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:35.912675 9181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a79977370da41839799f37406b2ad05" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36395 } }
I20260928 15:55:35.913344 9181 leader_election.cc:302] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a79977370da41839799f37406b2ad05; no voters:
I20260928 15:55:35.913992 9181 leader_election.cc:288] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:35.914151 9186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:35.914963 9186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [term 1 LEADER]: Becoming Leader. State: Replica: 1a79977370da41839799f37406b2ad05, State: Running, Role: LEADER
I20260928 15:55:35.915562 9186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a79977370da41839799f37406b2ad05" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36395 } }
I20260928 15:55:35.916472 9181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:35.918840 9188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a79977370da41839799f37406b2ad05. Latest consensus state: current_term: 1 leader_uuid: "1a79977370da41839799f37406b2ad05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a79977370da41839799f37406b2ad05" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36395 } } }
I20260928 15:55:35.919265 9188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:35.918962 9187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a79977370da41839799f37406b2ad05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a79977370da41839799f37406b2ad05" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36395 } } }
I20260928 15:55:35.919694 9187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:35.921141 9191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:35.928249 9191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:35.939184 9191 catalog_manager.cc:1347] Generated new cluster ID: 84358291d9424bfc89da4e616be55d9b
I20260928 15:55:35.939330 9191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:35.958813 9191 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:35.960855 9191 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:35.968994 9191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05: Generated new TSK 0
I20260928 15:55:35.970139 9191 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:55:36.044934 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36395
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:36.396518 9207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:36.396929 9207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:36.397105 9207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:36.404637 9207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:36.404839 9207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:36.405030 9207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:55:36.415283 9207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:36.416796 9207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:36.418390 9207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:36.433523 9213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:36.433619 9215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:36.433562 9212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:36.435014 9207 server_base.cc:1034] running on GCE node
I20260928 15:55:36.435740 9207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:36.436877 9207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:36.438122 9207 hybrid_clock.cc:648] HybridClock initialized: now 1790610936438061 us; error 56 us; skew 500 ppm
I20260928 15:55:36.438476 9207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:36 dist-test-slave-1vw0 krb5kdc[9101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610936, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:36.442061 9207 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:55:36.443612 9207 webserver.cc:460] Webserver started at http://127.22.157.193:38343/ using document root <none> and password file <none>
I20260928 15:55:36.444362 9207 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:36.444522 9207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:36.444912 9207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:36.447985 9207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "e36870abf9b54b459d4f3d0d11af229c"
format_stamp: "Formatted at 2026-09-28 15:55:36 on dist-test-slave-1vw0"
server_key: "8849b65023410e7153a273aa350bd6e4"
server_key_iv: "2d293c2aa0a8455a6238302545df2085"
server_key_version: "encryptionkey@0"
I20260928 15:55:36.448908 9207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "e36870abf9b54b459d4f3d0d11af229c"
format_stamp: "Formatted at 2026-09-28 15:55:36 on dist-test-slave-1vw0"
server_key: "8849b65023410e7153a273aa350bd6e4"
server_key_iv: "2d293c2aa0a8455a6238302545df2085"
server_key_version: "encryptionkey@0"
I20260928 15:55:36.454661 9207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:55:36.458549 9222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:36.460196 9207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:55:36.460423 9207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "e36870abf9b54b459d4f3d0d11af229c"
format_stamp: "Formatted at 2026-09-28 15:55:36 on dist-test-slave-1vw0"
server_key: "8849b65023410e7153a273aa350bd6e4"
server_key_iv: "2d293c2aa0a8455a6238302545df2085"
server_key_version: "encryptionkey@0"
I20260928 15:55:36.460690 9207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:36.494963 9207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:36.510643 9207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:36.510975 9207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:36.512084 9207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:36.513612 9207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:36.513715 9207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:36.513870 9207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:36.513940 9207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:36.549525 9207 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:38491
I20260928 15:55:36.549618 9334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:38491 every 8 connection(s)
I20260928 15:55:36.550885 9207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:55:36.556860 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 9207
I20260928 15:55:36.557049 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:55:36.557505 23159 external_mini_cluster.cc:1428] Setting key a2639c7a096b245b798859801f21fcce
Sep 28 15:55:36 dist-test-slave-1vw0 krb5kdc[9101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610936, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:36.572209 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:36.553591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:39677 (local address 127.22.157.254:36395)
0928 15:55:36.554086 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:36.554091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:36.555183 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 15:55:36.557260 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:36.557265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:36.557360 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:36.557485 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:36.559067 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:36.559809 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:36.561512 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:36.561930 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:36.562076 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:36.566261 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:36.566266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:36.566269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:36.566304 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:36.569029 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:36.569837 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:36.569847 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:36.569849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:36.569970 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:36.570420 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:36.570427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:36.570435 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:36.570708 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:36.570867 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:36.571594 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 15:55:36.571838 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
I20260928 15:55:36.573344 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:36.554214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:36395 (local address 127.22.157.193:39677)
0928 15:55:36.554900 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:36.554940 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:36.556901 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:36.557750 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:36.557759 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:36.558310 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:36.558891 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:36.558909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:36.559981 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:36.559986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:36.561361 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:36.561373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:36.562188 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:36.562197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:36.562442 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:36.563159 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:55:36.563183 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:36.566015 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:36.569184 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:36.569197 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:36.569218 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:36.569690 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:36.570092 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:36.570097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:36.570100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:36.570297 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:36.570840 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:36.570848 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:36.571207 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 15:55:36.571479 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 15:55:36.572049 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":283,"threads_started":1}
I20260928 15:55:36.574930 9335 heartbeater.cc:347] Connected to a master server at 127.22.157.254:36395
I20260928 15:55:36.575278 9335 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:36.576185 9335 heartbeater.cc:511] Master 127.22.157.254:36395 requested a full tablet report, sending...
I20260928 15:55:36.578540 9146 ts_manager.cc:194] Registered new tserver with Master: e36870abf9b54b459d4f3d0d11af229c (127.22.157.193:38491)
I20260928 15:55:36.580005 9146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:39677
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:55:36.622382 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36395
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:36.967593 9341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:36.967885 9341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:36.968042 9341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:36.975433 9341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:36.975560 9341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:36.975754 9341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:55:36.985574 9341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:36.987030 9341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:36.988407 9341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:37.003397 9346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:37.003315 9349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:37.003357 9347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:37.004560 9341 server_base.cc:1034] running on GCE node
I20260928 15:55:37.005352 9341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:37.006398 9341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:37.007692 9341 hybrid_clock.cc:648] HybridClock initialized: now 1790610937007621 us; error 69 us; skew 500 ppm
I20260928 15:55:37.008069 9341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:37 dist-test-slave-1vw0 krb5kdc[9101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:37.011878 9341 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:55:37.013585 9341 webserver.cc:460] Webserver started at http://127.22.157.194:40607/ using document root <none> and password file <none>
I20260928 15:55:37.014333 9341 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:37.014447 9341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:37.014753 9341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:37.018378 9341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "b3cd557acb654b649ad34c8ad8767599"
format_stamp: "Formatted at 2026-09-28 15:55:37 on dist-test-slave-1vw0"
server_key: "3b64b4d1ec62d5c6c8875bb623d1034b"
server_key_iv: "189e09116def164cf80a7145daf802fb"
server_key_version: "encryptionkey@0"
I20260928 15:55:37.019248 9341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "b3cd557acb654b649ad34c8ad8767599"
format_stamp: "Formatted at 2026-09-28 15:55:37 on dist-test-slave-1vw0"
server_key: "3b64b4d1ec62d5c6c8875bb623d1034b"
server_key_iv: "189e09116def164cf80a7145daf802fb"
server_key_version: "encryptionkey@0"
I20260928 15:55:37.025365 9341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260928 15:55:37.029899 9356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:37.031672 9341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 15:55:37.031921 9341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "b3cd557acb654b649ad34c8ad8767599"
format_stamp: "Formatted at 2026-09-28 15:55:37 on dist-test-slave-1vw0"
server_key: "3b64b4d1ec62d5c6c8875bb623d1034b"
server_key_iv: "189e09116def164cf80a7145daf802fb"
server_key_version: "encryptionkey@0"
I20260928 15:55:37.032173 9341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:37.063814 9341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:37.068338 9341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:37.068647 9341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:37.069718 9341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:37.071110 9341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:37.071215 9341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:37.071369 9341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:37.071452 9341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:37.106331 9341 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:36521
I20260928 15:55:37.106426 9468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:36521 every 8 connection(s)
I20260928 15:55:37.107705 9341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:55:37.112798 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 9341
I20260928 15:55:37.113010 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:55:37.113582 23159 external_mini_cluster.cc:1428] Setting key 114e9efbc648ffece2ad719c09fb2961
Sep 28 15:55:37 dist-test-slave-1vw0 krb5kdc[9101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:37.133296 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.110675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:45333 (local address 127.22.157.254:36395)
0928 15:55:37.111143 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:37.111147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:37.112219 (+ 1072us) server_negotiation.cc:407] Connection header received
0928 15:55:37.114813 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:37.114819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:37.114916 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:37.115037 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:37.117153 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.118138 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.120853 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.121346 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.121460 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.126419 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:37.126427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:37.126434 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:37.126497 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:37.129608 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:37.130504 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:37.130508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:37.130547 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:37.130702 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:37.131267 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:37.131272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:37.131276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:37.131583 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:37.131748 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:37.132624 (+ 876us) server_negotiation.cc:299] Negotiation successful
0928 15:55:37.132893 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":232,"threads_started":1}
I20260928 15:55:37.134378 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.111316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:36395 (local address 127.22.157.194:45333)
0928 15:55:37.112048 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:37.112094 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:37.114515 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:37.115300 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:37.115310 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:37.115996 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:37.116919 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.116950 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.118296 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.118304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.120603 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.120623 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.121616 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.121626 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.121993 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.122801 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:55:37.122842 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:37.126103 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:37.129785 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:37.129796 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:37.129815 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:37.130349 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:37.130828 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:37.130833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:37.130837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:37.131138 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:37.131728 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:37.131736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:37.132191 (+ 455us) client_negotiation.cc:769] Sending connection context
0928 15:55:37.132517 (+ 326us) client_negotiation.cc:240] Negotiation successful
0928 15:55:37.133150 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":300,"threads_started":1}
I20260928 15:55:37.136198 9469 heartbeater.cc:347] Connected to a master server at 127.22.157.254:36395
I20260928 15:55:37.136541 9469 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:37.137498 9469 heartbeater.cc:511] Master 127.22.157.254:36395 requested a full tablet report, sending...
I20260928 15:55:37.139169 9146 ts_manager.cc:194] Registered new tserver with Master: b3cd557acb654b649ad34c8ad8767599 (127.22.157.194:36521)
I20260928 15:55:37.140519 9146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:45333
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:55:37.177525 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36395
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:37.521171 9475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:37.521474 9475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:37.521569 9475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:37.529944 9475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:37.530068 9475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:37.530190 9475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:55:37.541018 9475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:37.542385 9475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:37.543733 9475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:37.559674 9480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:37.559700 9483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:37.559800 9481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:37.560647 9475 server_base.cc:1034] running on GCE node
I20260928 15:55:37.561306 9475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:37.562284 9475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:37.563578 9475 hybrid_clock.cc:648] HybridClock initialized: now 1790610937563496 us; error 68 us; skew 500 ppm
I20260928 15:55:37.563925 9475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:37 dist-test-slave-1vw0 krb5kdc[9101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:37.567343 9475 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:55:37.568894 9475 webserver.cc:460] Webserver started at http://127.22.157.195:45759/ using document root <none> and password file <none>
I20260928 15:55:37.569847 9475 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:37.570003 9475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:37.570688 9475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:37.576560 9475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "87aab4e0bcca455392ee00d0d9c50647"
format_stamp: "Formatted at 2026-09-28 15:55:37 on dist-test-slave-1vw0"
server_key: "203a7561a1c8030aa0b3558cfab2efc5"
server_key_iv: "02a16a1ac73c61f5a5b47dd939365e5c"
server_key_version: "encryptionkey@0"
I20260928 15:55:37.578284 9475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "87aab4e0bcca455392ee00d0d9c50647"
format_stamp: "Formatted at 2026-09-28 15:55:37 on dist-test-slave-1vw0"
server_key: "203a7561a1c8030aa0b3558cfab2efc5"
server_key_iv: "02a16a1ac73c61f5a5b47dd939365e5c"
server_key_version: "encryptionkey@0"
I20260928 15:55:37.583978 9335 heartbeater.cc:503] Master 127.22.157.254:36395 was elected leader, sending a full tablet report...
I20260928 15:55:37.588047 9475 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20260928 15:55:37.593750 9490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:37.595902 9475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 15:55:37.596143 9475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "87aab4e0bcca455392ee00d0d9c50647"
format_stamp: "Formatted at 2026-09-28 15:55:37 on dist-test-slave-1vw0"
server_key: "203a7561a1c8030aa0b3558cfab2efc5"
server_key_iv: "02a16a1ac73c61f5a5b47dd939365e5c"
server_key_version: "encryptionkey@0"
I20260928 15:55:37.596472 9475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:37.634686 9475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:37.640027 9475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:37.640305 9475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:37.641496 9475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:37.643015 9475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:37.643097 9475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:37.643186 9475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:37.643250 9475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:37.678215 9475 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:44673
I20260928 15:55:37.678294 9602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:44673 every 8 connection(s)
I20260928 15:55:37.679709 9475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:55:37.687345 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 9475
I20260928 15:55:37.687561 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:55:37.688128 23159 external_mini_cluster.cc:1428] Setting key 0a105f4b8be229208a997fa6d098c5ef
Sep 28 15:55:37 dist-test-slave-1vw0 krb5kdc[9101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610937, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:37.702739 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.682522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:45389 (local address 127.22.157.254:36395)
0928 15:55:37.683052 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:37.683056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:37.683948 (+ 892us) server_negotiation.cc:407] Connection header received
0928 15:55:37.685996 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:37.686001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:37.686117 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:37.686246 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:37.688466 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.689640 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.691840 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.692271 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.692374 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.696140 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:37.696145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:37.696149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:37.696186 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:37.699035 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:37.700083 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:37.700087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:37.700090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:37.700221 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:37.700697 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:37.700701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:37.700707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:37.701060 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:37.701239 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:37.702011 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 15:55:37.702287 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":292,"threads_started":1}
I20260928 15:55:37.704113 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.682976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:36395 (local address 127.22.157.195:45389)
0928 15:55:37.683758 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:37.683802 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:37.685679 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:37.686518 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:37.686528 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:37.687317 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:37.688228 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.688259 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.689813 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.689819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.691691 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.691706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.692473 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.692478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.692695 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.693479 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:55:37.693504 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:37.695907 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:37.699186 (+ 3279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:37.699194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:37.699209 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:37.699913 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:37.700340 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:37.700345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:37.700349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:37.700565 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:37.701199 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:37.701208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:37.701594 (+ 386us) client_negotiation.cc:769] Sending connection context
0928 15:55:37.701922 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 15:55:37.702536 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":291,"threads_started":1}
I20260928 15:55:37.705660 9603 heartbeater.cc:347] Connected to a master server at 127.22.157.254:36395
I20260928 15:55:37.706013 9603 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:37.706828 9603 heartbeater.cc:511] Master 127.22.157.254:36395 requested a full tablet report, sending...
I20260928 15:55:37.708473 9146 ts_manager.cc:194] Registered new tserver with Master: 87aab4e0bcca455392ee00d0d9c50647 (127.22.157.195:44673)
I20260928 15:55:37.709455 9146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:45389
I20260928 15:55:37.716840 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:55:37.737401 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.721314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.22.157.254:36395)
0928 15:55:37.721509 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:37.721514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:37.722076 (+ 562us) server_negotiation.cc:407] Connection header received
0928 15:55:37.722220 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:37.722225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:37.722299 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:37.722394 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:37.723942 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.725147 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.726803 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.727292 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.727407 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.730686 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:37.730693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:37.730697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:37.730743 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:37.733799 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:37.734754 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:37.734759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:37.734761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:37.734902 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:37.735402 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:37.735406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:37.735408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:37.735832 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:37.736208 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:37.736788 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 15:55:37.737033 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:55:37.742484 9146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:55:37.750694 9146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:55:37.791754 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.768136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.22.157.194:36521)
0928 15:55:37.781573 (+ 13437us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:37.781583 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:37.781627 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:55:37.781746 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:37.781755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:37.782061 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:37.783050 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:37.783318 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.784844 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.788915 (+ 4071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.790330 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.790537 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.790748 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:37.791200 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 15:55:37.791396 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13277,"thread_start_us":13136,"threads_started":1}
I20260928 15:55:37.792052 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.767782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:36521 (local address 127.0.0.1:45584)
0928 15:55:37.780419 (+ 12637us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:37.780451 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:37.780694 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:37.782332 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:37.782338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:37.782373 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:37.782807 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.782821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.785008 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.785013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.787557 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.787571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.790599 (+ 3028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.790607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.790850 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.790927 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 15:55:37.791098 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 15:55:37.791345 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12416,"mutex_wait_us":28,"thread_start_us":12973,"threads_started":1}
I20260928 15:55:37.793712 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.765346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:38491 (local address 127.0.0.1:38940)
0928 15:55:37.767284 (+ 1938us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:37.767330 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:37.767509 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:37.781746 (+ 14237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:37.781754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:37.781798 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:37.782297 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.782317 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.784835 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.784840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.789617 (+ 4777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.789637 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.791927 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.791935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.792899 (+ 964us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.792932 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 15:55:37.793029 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 15:55:37.793188 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1680,"mutex_wait_us":2231,"thread_start_us":1526,"threads_started":1}
I20260928 15:55:37.794217 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.765740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.22.157.193:38491)
0928 15:55:37.779568 (+ 13828us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:37.779586 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:37.779643 (+ 57us) server_negotiation.cc:407] Connection header received
0928 15:55:37.779771 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:37.779783 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:37.780161 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:37.782559 (+ 2398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:37.782855 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.784591 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.789861 (+ 5270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.791242 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.791441 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.791642 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:37.793640 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0928 15:55:37.793872 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13618,"thread_start_us":13432,"threads_started":1}
I20260928 15:55:37.795432 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.766958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:44673 (local address 127.0.0.1:47286)
0928 15:55:37.777156 (+ 10198us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:37.777185 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:37.777364 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:37.787156 (+ 9792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:37.787161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:37.787192 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:37.787655 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.787669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.789796 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.789801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.792306 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.792327 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.794588 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.794597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.794772 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.794808 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:55:37.794912 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 15:55:37.795047 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10025,"mutex_wait_us":1,"thread_start_us":9803,"threads_started":1}
I20260928 15:55:37.795485 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.767862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.22.157.195:44673)
0928 15:55:37.786138 (+ 18276us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:37.786147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:37.786184 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:55:37.786301 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:37.786310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:37.786595 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:37.786824 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:37.787860 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.789597 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.792557 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.794363 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.794579 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.794772 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:37.794977 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 15:55:37.795139 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18060,"thread_start_us":17874,"threads_started":1}
I20260928 15:55:37.796707 9404 tablet_service.cc:1467] Processing CreateTablet for tablet 47cf8af790e34cd592d1d098b2a444d5 (DEFAULT_TABLE table=test-table [id=2f5a93cf32824dda932891343b3aeb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:37.798450 9404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cf8af790e34cd592d1d098b2a444d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:37.798237 9270 tablet_service.cc:1467] Processing CreateTablet for tablet 47cf8af790e34cd592d1d098b2a444d5 (DEFAULT_TABLE table=test-table [id=2f5a93cf32824dda932891343b3aeb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:37.799561 9538 tablet_service.cc:1467] Processing CreateTablet for tablet 47cf8af790e34cd592d1d098b2a444d5 (DEFAULT_TABLE table=test-table [id=2f5a93cf32824dda932891343b3aeb2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:37.800124 9270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cf8af790e34cd592d1d098b2a444d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:37.801007 9538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cf8af790e34cd592d1d098b2a444d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:37.813828 9622 tablet_bootstrap.cc:492] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: Bootstrap starting.
I20260928 15:55:37.816182 9623 tablet_bootstrap.cc:492] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: Bootstrap starting.
I20260928 15:55:37.818216 9624 tablet_bootstrap.cc:492] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: Bootstrap starting.
I20260928 15:55:37.818514 9622 tablet_bootstrap.cc:654] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:37.820130 9622 log.cc:824] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:37.820317 9623 tablet_bootstrap.cc:654] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:37.821627 9623 log.cc:824] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:37.821732 9624 tablet_bootstrap.cc:654] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:37.823103 9624 log.cc:824] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:37.823869 9622 tablet_bootstrap.cc:492] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: No bootstrap required, opened a new log
I20260928 15:55:37.824241 9622 ts_tablet_manager.cc:1397] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260928 15:55:37.825599 9623 tablet_bootstrap.cc:492] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: No bootstrap required, opened a new log
I20260928 15:55:37.825918 9623 ts_tablet_manager.cc:1397] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260928 15:55:37.827862 9624 tablet_bootstrap.cc:492] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: No bootstrap required, opened a new log
I20260928 15:55:37.828173 9624 ts_tablet_manager.cc:1397] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 15:55:37.833180 9622 raft_consensus.cc:348] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } }
I20260928 15:55:37.833626 9622 raft_consensus.cc:374] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:37.833724 9622 raft_consensus.cc:729] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36870abf9b54b459d4f3d0d11af229c, State: Initialized, Role: FOLLOWER
I20260928 15:55:37.834518 9622 consensus_queue.cc:260] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } }
I20260928 15:55:37.837531 9623 raft_consensus.cc:348] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } }
I20260928 15:55:37.838032 9623 raft_consensus.cc:374] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:37.838068 9622 ts_tablet_manager.cc:1428] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260928 15:55:37.838125 9623 raft_consensus.cc:729] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cd557acb654b649ad34c8ad8767599, State: Initialized, Role: FOLLOWER
I20260928 15:55:37.838815 9623 consensus_queue.cc:260] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } }
I20260928 15:55:37.840042 9469 heartbeater.cc:503] Master 127.22.157.254:36395 was elected leader, sending a full tablet report...
I20260928 15:55:37.840090 9624 raft_consensus.cc:348] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } }
I20260928 15:55:37.840553 9624 raft_consensus.cc:374] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:37.840659 9624 raft_consensus.cc:729] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87aab4e0bcca455392ee00d0d9c50647, State: Initialized, Role: FOLLOWER
I20260928 15:55:37.840849 9623 ts_tablet_manager.cc:1428] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260928 15:55:37.841430 9624 consensus_queue.cc:260] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } }
I20260928 15:55:37.842568 9603 heartbeater.cc:503] Master 127.22.157.254:36395 was elected leader, sending a full tablet report...
I20260928 15:55:37.843297 9624 ts_tablet_manager.cc:1428] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
W20260928 15:55:37.860005 9470 tablet.cc:2369] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:55:37.916958 9630 raft_consensus.cc:482] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:55:37.917385 9630 raft_consensus.cc:504] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } }
I20260928 15:55:37.919504 9630 leader_election.cc:288] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e36870abf9b54b459d4f3d0d11af229c (127.22.157.193:38491), b3cd557acb654b649ad34c8ad8767599 (127.22.157.194:36521)
I20260928 15:55:37.926225 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.920107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:38491 (local address 127.22.157.195:43753)
0928 15:55:37.920426 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:37.920445 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:37.920636 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:37.921161 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:37.921169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:37.921202 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:37.921416 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.921430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.922728 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.922734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.924055 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.924067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.925364 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.925370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.925568 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.925596 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 15:55:37.925778 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 15:55:37.925929 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260928 15:55:37.926208 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.920081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:43753 (local address 127.22.157.193:38491)
0928 15:55:37.920394 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:37.920401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:37.920547 (+ 146us) server_negotiation.cc:407] Connection header received
0928 15:55:37.920865 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:37.920876 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:37.920976 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:37.921115 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:37.921581 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.922565 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.924237 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.925183 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.925362 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.925440 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:37.925807 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 15:55:37.925945 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 15:55:37.927654 9290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cf8af790e34cd592d1d098b2a444d5" candidate_uuid: "87aab4e0bcca455392ee00d0d9c50647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36870abf9b54b459d4f3d0d11af229c" is_pre_election: true
I20260928 15:55:37.928045 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.920859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:36521 (local address 127.22.157.195:42583)
0928 15:55:37.921820 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:37.921836 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:37.921951 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:37.922282 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:37.922285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:37.922305 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:37.922626 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.922635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.923809 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.923822 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.925732 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:37.925746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.927460 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.927464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:37.927589 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.927616 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 15:55:37.927697 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 15:55:37.927794 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":643,"threads_started":1}
I20260928 15:55:37.928244 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.921060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:42583 (local address 127.22.157.194:36521)
0928 15:55:37.921304 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:37.921310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:37.921941 (+ 631us) server_negotiation.cc:407] Connection header received
0928 15:55:37.922070 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:37.922075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:37.922159 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:37.922251 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:37.922812 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.923641 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.925916 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.927266 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.927451 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.927553 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:37.927771 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 15:55:37.927956 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 15:55:37.928443 9290 raft_consensus.cc:2454] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87aab4e0bcca455392ee00d0d9c50647 in term 0.
I20260928 15:55:37.929314 9491 leader_election.cc:302] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87aab4e0bcca455392ee00d0d9c50647, e36870abf9b54b459d4f3d0d11af229c; no voters:
I20260928 15:55:37.929515 9424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cf8af790e34cd592d1d098b2a444d5" candidate_uuid: "87aab4e0bcca455392ee00d0d9c50647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cd557acb654b649ad34c8ad8767599" is_pre_election: true
I20260928 15:55:37.929733 9630 raft_consensus.cc:2781] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:55:37.929908 9630 raft_consensus.cc:482] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:55:37.930048 9630 raft_consensus.cc:3037] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:37.930107 9424 raft_consensus.cc:2454] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87aab4e0bcca455392ee00d0d9c50647 in term 0.
W20260928 15:55:37.931418 9604 tablet.cc:2369] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:55:37.933360 9630 raft_consensus.cc:504] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } }
I20260928 15:55:37.934078 9630 leader_election.cc:288] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [CANDIDATE]: Term 1 election: Requested vote from peers e36870abf9b54b459d4f3d0d11af229c (127.22.157.193:38491), b3cd557acb654b649ad34c8ad8767599 (127.22.157.194:36521)
I20260928 15:55:37.934687 9290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cf8af790e34cd592d1d098b2a444d5" candidate_uuid: "87aab4e0bcca455392ee00d0d9c50647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e36870abf9b54b459d4f3d0d11af229c"
I20260928 15:55:37.934742 9424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cf8af790e34cd592d1d098b2a444d5" candidate_uuid: "87aab4e0bcca455392ee00d0d9c50647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3cd557acb654b649ad34c8ad8767599"
I20260928 15:55:37.934928 9290 raft_consensus.cc:3037] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:37.934958 9424 raft_consensus.cc:3037] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:37.937553 9424 raft_consensus.cc:2454] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87aab4e0bcca455392ee00d0d9c50647 in term 1.
I20260928 15:55:37.937609 9290 raft_consensus.cc:2454] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87aab4e0bcca455392ee00d0d9c50647 in term 1.
I20260928 15:55:37.938052 9491 leader_election.cc:302] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87aab4e0bcca455392ee00d0d9c50647, b3cd557acb654b649ad34c8ad8767599; no voters:
I20260928 15:55:37.938385 9630 raft_consensus.cc:2781] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:37.938872 9630 raft_consensus.cc:686] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 1 LEADER]: Becoming Leader. State: Replica: 87aab4e0bcca455392ee00d0d9c50647, State: Running, Role: LEADER
I20260928 15:55:37.939482 9630 consensus_queue.cc:237] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } }
I20260928 15:55:37.944516 9145 catalog_manager.cc:5495] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87aab4e0bcca455392ee00d0d9c50647 (127.22.157.195). New cstate: current_term: 1 leader_uuid: "87aab4e0bcca455392ee00d0d9c50647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87aab4e0bcca455392ee00d0d9c50647" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 44673 } health_report { overall_health: HEALTHY } } }
I20260928 15:55:37.991771 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:37.986013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.22.157.195:44673)
0928 15:55:37.986219 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:37.986224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:37.986243 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:55:37.986339 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:37.986344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:37.986440 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:37.986599 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:55:37.986987 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.987742 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.989367 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:37.989825 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:37.989986 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:37.990598 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:55:37.991185 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:55:37.991329 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:37.991511 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 15:55:37.991555 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:55:38.001027 9424 raft_consensus.cc:1261] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [term 1 FOLLOWER]: Refusing update from remote peer 87aab4e0bcca455392ee00d0d9c50647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:38.001068 9290 raft_consensus.cc:1261] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [term 1 FOLLOWER]: Refusing update from remote peer 87aab4e0bcca455392ee00d0d9c50647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:38.002110 9632 consensus_queue.cc:1035] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e36870abf9b54b459d4f3d0d11af229c" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:38.002614 9630 consensus_queue.cc:1035] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3cd557acb654b649ad34c8ad8767599" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:38.032254 9146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:38.032636 9146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:38.037457 9146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a79977370da41839799f37406b2ad05: Sending DeleteTablet for 3 replicas of tablet 47cf8af790e34cd592d1d098b2a444d5
I20260928 15:55:38.038960 9270 tablet_service.cc:1514] Processing DeleteTablet for tablet 47cf8af790e34cd592d1d098b2a444d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:38 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:38940
I20260928 15:55:38.038987 9404 tablet_service.cc:1514] Processing DeleteTablet for tablet 47cf8af790e34cd592d1d098b2a444d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:38 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:45584
I20260928 15:55:38.039438 9538 tablet_service.cc:1514] Processing DeleteTablet for tablet 47cf8af790e34cd592d1d098b2a444d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:38 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:47286
I20260928 15:55:38.044520 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 9207
I20260928 15:55:38.049054 9646 tablet_replica.cc:330] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: stopping tablet replica
I20260928 15:55:38.049526 9646 raft_consensus.cc:2229] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:55:38.050271 9646 raft_consensus.cc:2258] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:38.055341 9646 ts_tablet_manager.cc:1905] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:38.065272 9645 tablet_replica.cc:330] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: stopping tablet replica
I20260928 15:55:38.065847 9645 raft_consensus.cc:2229] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:38.066459 9645 raft_consensus.cc:2258] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:38.068971 9645 ts_tablet_manager.cc:1905] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 15:55:38 dist-test-slave-1vw0 krb5kdc[9101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:55:38.072903 9644 tablet_replica.cc:330] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: stopping tablet replica
I20260928 15:55:38.073496 9644 raft_consensus.cc:2229] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:38.074141 9644 raft_consensus.cc:2258] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:38.074740 9646 ts_tablet_manager.cc:1918] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:38.074936 9646 log.cc:1196] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/47cf8af790e34cd592d1d098b2a444d5
I20260928 15:55:38.075519 9646 ts_tablet_manager.cc:1939] T 47cf8af790e34cd592d1d098b2a444d5 P 87aab4e0bcca455392ee00d0d9c50647: Deleting consensus metadata
I20260928 15:55:38.076625 9645 ts_tablet_manager.cc:1918] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:38.076843 9645 log.cc:1196] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/47cf8af790e34cd592d1d098b2a444d5
I20260928 15:55:38.077404 9645 ts_tablet_manager.cc:1939] T 47cf8af790e34cd592d1d098b2a444d5 P e36870abf9b54b459d4f3d0d11af229c: Deleting consensus metadata
I20260928 15:55:38.077621 9132 catalog_manager.cc:4841] TS 87aab4e0bcca455392ee00d0d9c50647 (127.22.157.195:44673): tablet 47cf8af790e34cd592d1d098b2a444d5 (table test-table [id=2f5a93cf32824dda932891343b3aeb2d]) successfully deleted
I20260928 15:55:38.078169 9644 ts_tablet_manager.cc:1905] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:38.079190 9130 catalog_manager.cc:4841] TS e36870abf9b54b459d4f3d0d11af229c (127.22.157.193:38491): tablet 47cf8af790e34cd592d1d098b2a444d5 (table test-table [id=2f5a93cf32824dda932891343b3aeb2d]) successfully deleted
I20260928 15:55:38.083205 9644 ts_tablet_manager.cc:1918] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:38.083411 9644 log.cc:1196] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/47cf8af790e34cd592d1d098b2a444d5
I20260928 15:55:38.083966 9644 ts_tablet_manager.cc:1939] T 47cf8af790e34cd592d1d098b2a444d5 P b3cd557acb654b649ad34c8ad8767599: Deleting consensus metadata
I20260928 15:55:38.085870 9130 catalog_manager.cc:4841] TS b3cd557acb654b649ad34c8ad8767599 (127.22.157.194:36521): tablet 47cf8af790e34cd592d1d098b2a444d5 (table test-table [id=2f5a93cf32824dda932891343b3aeb2d]) successfully deleted
I20260928 15:55:38.091739 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:38.046381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.22.157.193:38491)
0928 15:55:38.046618 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:38.046627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:38.059136 (+ 12509us) server_negotiation.cc:407] Connection header received
0928 15:55:38.059752 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:38.059761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:38.059890 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:38.059976 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:38.061967 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:38.063194 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:38.065502 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:38.066262 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:38.066390 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:38.083911 (+ 17521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:38.083933 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:38.083963 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:38.084018 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:38.088232 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:38.089276 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:38.089287 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:38.089291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:38.089488 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:38.089950 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:38.089956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:38.089959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:38.090484 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:38.090806 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:38.091101 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 15:55:38.091346 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:55:38.092557 9330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:38.186182 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 9207
I20260928 15:55:38.210685 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 9341
Sep 28 15:55:38 dist-test-slave-1vw0 krb5kdc[9101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:55:38.229283 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:38.211529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.22.157.194:36521)
0928 15:55:38.211721 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:38.211727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:38.211750 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:55:38.211894 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:38.211900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:38.211992 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:38.212099 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:38.213480 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:38.214659 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:38.216208 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:38.216990 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:38.217147 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:38.221881 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:38.221903 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:38.221926 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:38.221992 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:38.226094 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:38.226983 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:38.226993 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:38.226997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:38.227160 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:38.227516 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:38.227521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:38.227526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:38.228052 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:38.228355 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:38.228651 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 15:55:38.228922 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:55:38.230033 9464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:38.321686 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 9341
I20260928 15:55:38.346028 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 9475
Sep 28 15:55:38 dist-test-slave-1vw0 krb5kdc[9101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:55:38.366358 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:38.346922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.22.157.195:44673)
0928 15:55:38.347119 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:38.347124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:38.347142 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:55:38.347337 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:38.347341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:38.347407 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:38.347507 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:38.349407 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:38.350251 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:38.352162 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:38.352595 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:38.352697 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:38.358748 (+ 6051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:38.358760 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:38.358775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:38.358821 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:38.362566 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:38.363615 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:38.363624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:38.363626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:38.363844 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:38.364364 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:38.364369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:38.364371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:38.365094 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:38.365428 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:38.365775 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 15:55:38.365989 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:55:38.367214 9598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:38.494503 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 9475
I20260928 15:55:38.515442 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 9114
I20260928 15:55:38.516889 9176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:38.632057 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 9114
2026-09-28T15:55:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4652 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T15:55:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:38Z Disabled control of system clock
I20260928 15:55:38.720502 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:35881
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:35881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:55:39.085520 9655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:39.085850 9655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:39.085934 9655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:39.093461 9655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:39.093592 9655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:39.093640 9655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:39.093693 9655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:39.093739 9655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:39.103696 9655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:35881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:35881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:39.105119 9655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:39.106967 9655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:39.119493 9660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:39.119473 9661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:39.119323 9663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:39.120639 9655 server_base.cc:1034] running on GCE node
I20260928 15:55:39.121555 9655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:39.123183 9655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:39.124437 9655 hybrid_clock.cc:648] HybridClock initialized: now 1790610939124390 us; error 50 us; skew 500 ppm
I20260928 15:55:39.124923 9655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:39.127355 9655 webserver.cc:460] Webserver started at http://127.22.157.254:39471/ using document root <none> and password file <none>
I20260928 15:55:39.128121 9655 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:39.128268 9655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:39.128620 9655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:39.131769 9655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "ff7443faf6a145e888a37d229d9dcbad"
format_stamp: "Formatted at 2026-09-28 15:55:39 on dist-test-slave-1vw0"
server_key: "b77426915f6c5d6b8a5fddf75597e1c6"
server_key_iv: "d847ead0385d15e9b94c354f59b2d61f"
server_key_version: "encryptionkey@0"
I20260928 15:55:39.132658 9655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "ff7443faf6a145e888a37d229d9dcbad"
format_stamp: "Formatted at 2026-09-28 15:55:39 on dist-test-slave-1vw0"
server_key: "b77426915f6c5d6b8a5fddf75597e1c6"
server_key_iv: "d847ead0385d15e9b94c354f59b2d61f"
server_key_version: "encryptionkey@0"
I20260928 15:55:39.138427 9655 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:55:39.142488 9669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:39.144021 9655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:55:39.144241 9655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "ff7443faf6a145e888a37d229d9dcbad"
format_stamp: "Formatted at 2026-09-28 15:55:39 on dist-test-slave-1vw0"
server_key: "b77426915f6c5d6b8a5fddf75597e1c6"
server_key_iv: "d847ead0385d15e9b94c354f59b2d61f"
server_key_version: "encryptionkey@0"
I20260928 15:55:39.144481 9655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:39.161306 9655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:39.162220 9655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:39.162529 9655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:39.181911 9655 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:35881
I20260928 15:55:39.181917 9720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:35881 every 8 connection(s)
I20260928 15:55:39.183408 9655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:39.187309 9721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:39.191130 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 9655
I20260928 15:55:39.191314 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:39.191869 23159 external_mini_cluster.cc:1428] Setting key 9d5e0cbb75467741a075f7dd7fbdcbec
I20260928 15:55:39.200917 9721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad: Bootstrap starting.
I20260928 15:55:39.204460 9721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:39.205623 9721 log.cc:824] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:39.206841 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:39.193955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.22.157.254:35881)
0928 15:55:39.195301 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:39.195320 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:39.195396 (+ 76us) server_negotiation.cc:407] Connection header received
0928 15:55:39.197326 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:39.197353 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:39.197899 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:39.198343 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:39.198980 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:39.200306 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:39.201900 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:39.202774 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:39.203000 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:39.204222 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:39.204230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:39.204245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:39.204925 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:39.205036 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:39.205058 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:39.205192 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:39.205424 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 15:55:39.205792 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":447,"threads_started":1}
I20260928 15:55:39.208644 9721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad: No bootstrap required, opened a new log
I20260928 15:55:39.216022 9721 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7443faf6a145e888a37d229d9dcbad" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35881 } }
I20260928 15:55:39.216439 9721 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:39.216542 9721 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff7443faf6a145e888a37d229d9dcbad, State: Initialized, Role: FOLLOWER
I20260928 15:55:39.217207 9721 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7443faf6a145e888a37d229d9dcbad" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35881 } }
I20260928 15:55:39.217449 9721 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:39.217585 9721 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:39.217763 9721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:39.219977 9721 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7443faf6a145e888a37d229d9dcbad" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35881 } }
I20260928 15:55:39.220528 9721 leader_election.cc:302] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff7443faf6a145e888a37d229d9dcbad; no voters:
I20260928 15:55:39.221143 9721 leader_election.cc:288] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:39.221308 9726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:39.222004 9726 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [term 1 LEADER]: Becoming Leader. State: Replica: ff7443faf6a145e888a37d229d9dcbad, State: Running, Role: LEADER
I20260928 15:55:39.222604 9726 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7443faf6a145e888a37d229d9dcbad" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35881 } }
I20260928 15:55:39.223419 9721 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:39.225594 9728 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff7443faf6a145e888a37d229d9dcbad. Latest consensus state: current_term: 1 leader_uuid: "ff7443faf6a145e888a37d229d9dcbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7443faf6a145e888a37d229d9dcbad" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35881 } } }
I20260928 15:55:39.226119 9728 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:39.227090 9735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:39.226985 9727 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff7443faf6a145e888a37d229d9dcbad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff7443faf6a145e888a37d229d9dcbad" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 35881 } } }
I20260928 15:55:39.227526 9727 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:39.233667 9735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:39.243301 9735 catalog_manager.cc:1347] Generated new cluster ID: fbdc2cdb7e4a4be28c60b350138f1b26
I20260928 15:55:39.243434 9735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:39.258566 9735 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:39.259965 9735 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:39.269526 9735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad: Generated new TSK 0
I20260928 15:55:39.270466 9735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:55:39.281283 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35881
--builtin_ntp_servers=127.22.157.212:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:55:39.646893 9745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:39.647240 9745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:39.647399 9745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:39.655136 9745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:39.655293 9745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:39.655469 9745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:55:39.664963 9745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:39.666510 9745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:39.668435 9745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:39.682857 9753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:39.682948 9750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:39.682927 9751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:39.684029 9745 server_base.cc:1034] running on GCE node
I20260928 15:55:39.684821 9745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:39.686347 9745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:39.687724 9745 hybrid_clock.cc:648] HybridClock initialized: now 1790610939687647 us; error 66 us; skew 500 ppm
I20260928 15:55:39.688206 9745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:39.690760 9745 webserver.cc:460] Webserver started at http://127.22.157.193:46311/ using document root <none> and password file <none>
I20260928 15:55:39.691548 9745 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:39.691689 9745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:39.692040 9745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:39.695300 9745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "c471e68546434550a8a679797fe16e9b"
format_stamp: "Formatted at 2026-09-28 15:55:39 on dist-test-slave-1vw0"
server_key: "d167ca58045484848738d533896a1949"
server_key_iv: "7d389f348c5c5db24a4966bb7f6d2f3d"
server_key_version: "encryptionkey@0"
I20260928 15:55:39.696201 9745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "c471e68546434550a8a679797fe16e9b"
format_stamp: "Formatted at 2026-09-28 15:55:39 on dist-test-slave-1vw0"
server_key: "d167ca58045484848738d533896a1949"
server_key_iv: "7d389f348c5c5db24a4966bb7f6d2f3d"
server_key_version: "encryptionkey@0"
I20260928 15:55:39.702018 9745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260928 15:55:39.705879 9759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:39.707382 9745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:55:39.707584 9745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "c471e68546434550a8a679797fe16e9b"
format_stamp: "Formatted at 2026-09-28 15:55:39 on dist-test-slave-1vw0"
server_key: "d167ca58045484848738d533896a1949"
server_key_iv: "7d389f348c5c5db24a4966bb7f6d2f3d"
server_key_version: "encryptionkey@0"
I20260928 15:55:39.707818 9745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:39.729019 9745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:39.729936 9745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:39.730268 9745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:39.731395 9745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:39.733106 9745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:39.733227 9745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:39.733358 9745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:39.733436 9745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:39.767817 9745 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:37185
I20260928 15:55:39.767915 9872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:37185 every 8 connection(s)
I20260928 15:55:39.769254 9745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:55:39.774587 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 9745
I20260928 15:55:39.774799 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:55:39.775252 23159 external_mini_cluster.cc:1428] Setting key fb4de0722e7eaeaead12ff19a3403363
I20260928 15:55:39.778502 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35881
--builtin_ntp_servers=127.22.157.212:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:55:39.785599 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:39.771927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:57133 (local address 127.22.157.254:35881)
0928 15:55:39.772377 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:39.772384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:39.773775 (+ 1391us) server_negotiation.cc:407] Connection header received
0928 15:55:39.775773 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:39.775779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:39.775870 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:39.775968 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:39.777460 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:39.778505 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:39.780870 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:39.781622 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:39.781816 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:39.783710 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:39.783718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:39.783726 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:39.783753 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:39.783805 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:39.783823 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:39.783911 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:39.784995 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0928 15:55:39.785136 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260928 15:55:39.786948 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:39.772624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35881 (local address 127.22.157.193:57133)
0928 15:55:39.773617 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:39.773661 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:39.775482 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:39.776239 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:39.776248 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:39.776372 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:39.777254 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:39.777280 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:39.778718 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:39.778726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:39.780687 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:39.780706 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:39.781935 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:39.781945 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:39.782315 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:39.783269 (+ 954us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:55:39.783274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:39.783340 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:55:39.783472 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:39.784692 (+ 1220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:39.784705 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:39.784713 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 15:55:39.784938 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 15:55:39.785416 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":275,"threads_started":1}
I20260928 15:55:39.789877 9873 heartbeater.cc:347] Connected to a master server at 127.22.157.254:35881
I20260928 15:55:39.790257 9873 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:39.791172 9873 heartbeater.cc:511] Master 127.22.157.254:35881 requested a full tablet report, sending...
I20260928 15:55:39.793494 9686 ts_manager.cc:194] Registered new tserver with Master: c471e68546434550a8a679797fe16e9b (127.22.157.193:37185)
I20260928 15:55:39.795076 9686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.193:57133
W20260928 15:55:40.122922 9877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:40.123283 9877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:40.123486 9877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:40.131381 9877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:40.131532 9877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:40.131711 9877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:55:40.141284 9877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:40.142668 9877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:40.144493 9877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:40.161157 9885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:40.161119 9883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:40.161194 9882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:40.162349 9877 server_base.cc:1034] running on GCE node
I20260928 15:55:40.162973 9877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:40.163949 9877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:40.165153 9877 hybrid_clock.cc:648] HybridClock initialized: now 1790610940165079 us; error 63 us; skew 500 ppm
I20260928 15:55:40.165506 9877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:40.167973 9877 webserver.cc:460] Webserver started at http://127.22.157.194:44269/ using document root <none> and password file <none>
I20260928 15:55:40.168800 9877 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:40.168923 9877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:40.169255 9877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:40.172250 9877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3"
format_stamp: "Formatted at 2026-09-28 15:55:40 on dist-test-slave-1vw0"
server_key: "c6540de48dea94e56ff697a176989a86"
server_key_iv: "5efc6d0180c7c0a18e663965b06923d4"
server_key_version: "encryptionkey@0"
I20260928 15:55:40.173122 9877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3"
format_stamp: "Formatted at 2026-09-28 15:55:40 on dist-test-slave-1vw0"
server_key: "c6540de48dea94e56ff697a176989a86"
server_key_iv: "5efc6d0180c7c0a18e663965b06923d4"
server_key_version: "encryptionkey@0"
I20260928 15:55:40.178610 9877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:55:40.182560 9891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:40.184090 9877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:55:40.184263 9877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3"
format_stamp: "Formatted at 2026-09-28 15:55:40 on dist-test-slave-1vw0"
server_key: "c6540de48dea94e56ff697a176989a86"
server_key_iv: "5efc6d0180c7c0a18e663965b06923d4"
server_key_version: "encryptionkey@0"
I20260928 15:55:40.184468 9877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:40.214985 9877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:40.215884 9877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:40.216202 9877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:40.217476 9877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:40.219105 9877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:40.219197 9877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:40.219338 9877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:40.219429 9877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:40.252084 9877 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:41845
I20260928 15:55:40.252141 10003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:41845 every 8 connection(s)
I20260928 15:55:40.253453 9877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:55:40.262053 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 9877
I20260928 15:55:40.262229 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:55:40.262701 23159 external_mini_cluster.cc:1428] Setting key ec7e27cea7c0becf45dcbd8b5cb2b0ac
I20260928 15:55:40.266033 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35881
--builtin_ntp_servers=127.22.157.212:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:55:40.267369 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.256187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:49795 (local address 127.22.157.254:35881)
0928 15:55:40.256405 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:40.256411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:40.257885 (+ 1474us) server_negotiation.cc:407] Connection header received
0928 15:55:40.259658 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:40.259662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:40.259731 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:40.259820 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:40.260994 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.261798 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.263917 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.264418 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.264519 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.266361 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:40.266366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:40.266370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:40.266388 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:40.266421 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:40.266433 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:40.266490 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:40.266826 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 15:55:40.266951 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:55:40.268416 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.256710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35881 (local address 127.22.157.194:49795)
0928 15:55:40.257721 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:40.257767 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:40.259398 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:40.260060 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:40.260068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:40.260158 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:40.260842 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.260863 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.262009 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.262023 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.263769 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.263788 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.264694 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.264750 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.265113 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.265929 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:55:40.265932 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:40.265983 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:55:40.266076 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:40.266575 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:40.266583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:40.266589 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 15:55:40.266729 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 15:55:40.267085 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":294,"threads_started":1}
I20260928 15:55:40.270222 10004 heartbeater.cc:347] Connected to a master server at 127.22.157.254:35881
I20260928 15:55:40.270562 10004 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:40.271534 10004 heartbeater.cc:511] Master 127.22.157.254:35881 requested a full tablet report, sending...
I20260928 15:55:40.273325 9686 ts_manager.cc:194] Registered new tserver with Master: d0c2d9c4cb5b4389a478de6ce7ce53e3 (127.22.157.194:41845)
I20260928 15:55:40.274237 9686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.194:49795
W20260928 15:55:40.607795 10007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:40.608104 10007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:40.608256 10007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:40.615882 10007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:40.616009 10007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:40.616189 10007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:55:40.625774 10007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:35881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:40.627213 10007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:40.629140 10007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:40.643198 10012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:40.643221 10013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:40.643302 10015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:40.643924 10007 server_base.cc:1034] running on GCE node
I20260928 15:55:40.644706 10007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:40.645805 10007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:40.647042 10007 hybrid_clock.cc:648] HybridClock initialized: now 1790610940646999 us; error 45 us; skew 500 ppm
I20260928 15:55:40.647385 10007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:40.650137 10007 webserver.cc:460] Webserver started at http://127.22.157.195:45719/ using document root <none> and password file <none>
I20260928 15:55:40.650899 10007 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:40.651038 10007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:40.651372 10007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:40.654624 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "e230b025f0a74eddaaa31a4c3d895629"
format_stamp: "Formatted at 2026-09-28 15:55:40 on dist-test-slave-1vw0"
server_key: "0e33723db856a048d9a10168142a7e87"
server_key_iv: "e0c971b54d2f4cdd6494011ca3beeba9"
server_key_version: "encryptionkey@0"
I20260928 15:55:40.655468 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "e230b025f0a74eddaaa31a4c3d895629"
format_stamp: "Formatted at 2026-09-28 15:55:40 on dist-test-slave-1vw0"
server_key: "0e33723db856a048d9a10168142a7e87"
server_key_iv: "e0c971b54d2f4cdd6494011ca3beeba9"
server_key_version: "encryptionkey@0"
I20260928 15:55:40.661026 10007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:55:40.664644 10021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:40.666218 10007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:40.666390 10007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "e230b025f0a74eddaaa31a4c3d895629"
format_stamp: "Formatted at 2026-09-28 15:55:40 on dist-test-slave-1vw0"
server_key: "0e33723db856a048d9a10168142a7e87"
server_key_iv: "e0c971b54d2f4cdd6494011ca3beeba9"
server_key_version: "encryptionkey@0"
I20260928 15:55:40.666620 10007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:40.705327 10007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:40.706367 10007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:40.706696 10007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:40.707762 10007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:40.709452 10007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:40.709542 10007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:40.709648 10007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:40.709725 10007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:40.744328 10007 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:41051
I20260928 15:55:40.744421 10133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:41051 every 8 connection(s)
I20260928 15:55:40.745703 10007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:55:40.747367 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 10007
I20260928 15:55:40.747552 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:55:40.747990 23159 external_mini_cluster.cc:1428] Setting key 24195817927c8a62f38b2b423e0054ad
I20260928 15:55:40.760315 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.748829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:42749 (local address 127.22.157.254:35881)
0928 15:55:40.749062 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:40.749067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:40.750901 (+ 1834us) server_negotiation.cc:407] Connection header received
0928 15:55:40.752988 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:40.752994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:40.753098 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:40.753232 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:40.754612 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.755456 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.757283 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.757760 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.757902 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.759316 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:40.759323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:40.759332 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:40.759355 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:40.759389 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:40.759402 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:40.759507 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:40.759806 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 15:55:40.759920 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 15:55:40.761523 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.749577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:35881 (local address 127.22.157.195:42749)
0928 15:55:40.750743 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:40.750789 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:40.752647 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:40.753508 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:40.753523 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:40.753679 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:40.754403 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.754424 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.755650 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.755655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.757118 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.757132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.758017 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.758026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.758287 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.758957 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:55:40.758960 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:40.759003 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:55:40.759097 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:40.759506 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:40.759520 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:40.759531 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:55:40.759795 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 15:55:40.760251 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":371,"threads_started":1}
I20260928 15:55:40.762990 10134 heartbeater.cc:347] Connected to a master server at 127.22.157.254:35881
I20260928 15:55:40.763314 10134 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:40.764170 10134 heartbeater.cc:511] Master 127.22.157.254:35881 requested a full tablet report, sending...
I20260928 15:55:40.765625 9686 ts_manager.cc:194] Registered new tserver with Master: e230b025f0a74eddaaa31a4c3d895629 (127.22.157.195:41051)
I20260928 15:55:40.766487 9686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.195:42749
I20260928 15:55:40.776378 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:55:40.786842 9875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.780936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.22.157.254:35881)
0928 15:55:40.781163 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:40.781169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:40.781526 (+ 357us) server_negotiation.cc:407] Connection header received
0928 15:55:40.781665 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:40.781669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:40.781736 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:40.781829 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:40.782362 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.783148 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.784468 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.784980 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.785081 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.786189 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:40.786194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:40.786197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:40.786216 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:40.786249 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:40.786262 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:40.786391 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:40.786572 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 15:55:40.786617 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:55:40.791011 9686 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:55:40.797762 9686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:55:40.799098 9873 heartbeater.cc:503] Master 127.22.157.254:35881 was elected leader, sending a full tablet report...
I20260928 15:55:40.832878 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.811492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:41845 (local address 127.0.0.1:40350)
0928 15:55:40.822297 (+ 10805us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:40.822367 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:40.822617 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:40.824564 (+ 1947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:40.824572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:40.824602 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:40.825459 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.825480 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.827508 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.827516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.829857 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.829870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.831749 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.831755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.832025 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.832093 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 15:55:40.832241 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 15:55:40.832446 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10399,"mutex_wait_us":1,"thread_start_us":12792,"threads_started":1}
I20260928 15:55:40.833638 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.811818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37185 (local address 127.0.0.1:49660)
0928 15:55:40.823660 (+ 11842us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:40.823680 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:40.824080 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:40.825164 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:40.825169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:40.825202 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:40.825963 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.825981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.828129 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.828137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.830512 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.830528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.832627 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.832634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.832939 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.832989 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 15:55:40.833092 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 15:55:40.833225 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11696,"mutex_wait_us":1,"thread_start_us":9915,"threads_started":1}
I20260928 15:55:40.834084 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.811801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.22.157.194:41845)
0928 15:55:40.823301 (+ 11500us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:40.823313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:40.823352 (+ 39us) server_negotiation.cc:407] Connection header received
0928 15:55:40.823480 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:40.823492 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:40.823807 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:40.824059 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:40.825667 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.827386 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.830066 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.831554 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.831738 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.831991 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:40.833343 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0928 15:55:40.833564 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11316,"thread_start_us":11157,"threads_started":1}
I20260928 15:55:40.834640 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.812552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.22.157.193:37185)
0928 15:55:40.822529 (+ 9977us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:40.822539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:40.823793 (+ 1254us) server_negotiation.cc:407] Connection header received
0928 15:55:40.824307 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:40.824318 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:40.824599 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:40.824852 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:40.826160 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.827971 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.830768 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.832392 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.832627 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.832884 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:40.834004 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0928 15:55:40.834223 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9776,"thread_start_us":11408,"threads_started":1}
I20260928 15:55:40.835350 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.812445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:41051 (local address 127.0.0.1:46292)
0928 15:55:40.825648 (+ 13203us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:40.825668 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:40.825864 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:40.826799 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:40.826804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:40.826830 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:40.827340 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.827352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.829374 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.829380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.832038 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.832055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.834559 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.834566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.834735 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.834767 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 15:55:40.834857 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 15:55:40.834991 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13062,"mutex_wait_us":216,"thread_start_us":12745,"threads_started":1}
I20260928 15:55:40.836280 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.813211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.22.157.195:41051)
0928 15:55:40.824856 (+ 11645us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:40.824866 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:40.825792 (+ 926us) server_negotiation.cc:407] Connection header received
0928 15:55:40.826059 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:40.826070 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:40.826292 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:40.826453 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:40.827632 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.829214 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.832258 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.833989 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.834204 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.834410 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:40.835707 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0928 15:55:40.835909 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11416,"thread_start_us":11183,"threads_started":1}
I20260928 15:55:40.838449 9807 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4e393246db4ce19b1270dd3ef4fb43 (DEFAULT_TABLE table=test-table [id=08b064b58896444b956c150563cbe6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:40.838510 9939 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4e393246db4ce19b1270dd3ef4fb43 (DEFAULT_TABLE table=test-table [id=08b064b58896444b956c150563cbe6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:40.839831 9807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4e393246db4ce19b1270dd3ef4fb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:40.839884 9939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4e393246db4ce19b1270dd3ef4fb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:40.839715 10069 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4e393246db4ce19b1270dd3ef4fb43 (DEFAULT_TABLE table=test-table [id=08b064b58896444b956c150563cbe6c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:40.841146 10069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4e393246db4ce19b1270dd3ef4fb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:40.852685 10152 tablet_bootstrap.cc:492] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: Bootstrap starting.
I20260928 15:55:40.857100 10152 tablet_bootstrap.cc:654] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:40.857625 10154 tablet_bootstrap.cc:492] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: Bootstrap starting.
I20260928 15:55:40.858759 10152 log.cc:824] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:40.860396 10153 tablet_bootstrap.cc:492] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: Bootstrap starting.
I20260928 15:55:40.861855 10154 tablet_bootstrap.cc:654] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:40.862210 10152 tablet_bootstrap.cc:492] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: No bootstrap required, opened a new log
I20260928 15:55:40.862489 10152 ts_tablet_manager.cc:1397] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260928 15:55:40.863291 10154 log.cc:824] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:40.864593 10153 tablet_bootstrap.cc:654] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:40.866317 10153 log.cc:824] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:40.869604 10153 tablet_bootstrap.cc:492] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: No bootstrap required, opened a new log
I20260928 15:55:40.869901 10153 ts_tablet_manager.cc:1397] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 15:55:40.870052 10154 tablet_bootstrap.cc:492] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: No bootstrap required, opened a new log
I20260928 15:55:40.870334 10154 ts_tablet_manager.cc:1397] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260928 15:55:40.872023 10152 raft_consensus.cc:348] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } }
I20260928 15:55:40.872500 10152 raft_consensus.cc:374] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:40.872617 10152 raft_consensus.cc:729] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c471e68546434550a8a679797fe16e9b, State: Initialized, Role: FOLLOWER
I20260928 15:55:40.873270 10152 consensus_queue.cc:260] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } }
I20260928 15:55:40.875130 10152 ts_tablet_manager.cc:1428] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 15:55:40.880901 10153 raft_consensus.cc:348] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } }
I20260928 15:55:40.881366 10153 raft_consensus.cc:374] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:40.881492 10153 raft_consensus.cc:729] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c2d9c4cb5b4389a478de6ce7ce53e3, State: Initialized, Role: FOLLOWER
I20260928 15:55:40.881660 10154 raft_consensus.cc:348] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } }
I20260928 15:55:40.882046 10154 raft_consensus.cc:374] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:40.882140 10154 raft_consensus.cc:729] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230b025f0a74eddaaa31a4c3d895629, State: Initialized, Role: FOLLOWER
I20260928 15:55:40.882366 10153 consensus_queue.cc:260] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } }
I20260928 15:55:40.882764 10154 consensus_queue.cc:260] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } }
I20260928 15:55:40.883571 10004 heartbeater.cc:503] Master 127.22.157.254:35881 was elected leader, sending a full tablet report...
I20260928 15:55:40.883882 10134 heartbeater.cc:503] Master 127.22.157.254:35881 was elected leader, sending a full tablet report...
I20260928 15:55:40.884337 10153 ts_tablet_manager.cc:1428] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 15:55:40.884454 10154 ts_tablet_manager.cc:1428] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260928 15:55:40.951930 10158 raft_consensus.cc:482] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:55:40.952279 10158 raft_consensus.cc:504] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } }
I20260928 15:55:40.954007 10158 leader_election.cc:288] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0c2d9c4cb5b4389a478de6ce7ce53e3 (127.22.157.194:41845), e230b025f0a74eddaaa31a4c3d895629 (127.22.157.195:41051)
I20260928 15:55:40.961664 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.954697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:36563 (local address 127.22.157.195:41051)
0928 15:55:40.954953 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:40.954959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:40.956119 (+ 1160us) server_negotiation.cc:407] Connection header received
0928 15:55:40.956349 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:40.956358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:40.956417 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:40.956513 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:40.957165 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.957868 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.959933 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.960777 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.960947 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.961014 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:40.961308 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 15:55:40.961426 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 15:55:40.961670 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.954857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:44027 (local address 127.22.157.194:41845)
0928 15:55:40.955206 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:40.955214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:40.956088 (+ 874us) server_negotiation.cc:407] Connection header received
0928 15:55:40.956348 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:40.956358 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:40.956427 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:40.956526 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:40.957253 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.957948 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.959957 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.960693 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.960888 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.960970 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:40.961260 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 15:55:40.961385 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260928 15:55:40.961938 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.954307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:41845 (local address 127.22.157.193:44027)
0928 15:55:40.955973 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:40.955993 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:40.956202 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:40.956539 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:40.956543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:40.956565 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:40.957110 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.957118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.958089 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.958098 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.959740 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.959751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.960914 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.960920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.961089 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.961116 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 15:55:40.961222 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 15:55:40.961350 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1510,"mutex_wait_us":39,"spinlock_wait_cycles":54016,"thread_start_us":1406,"threads_started":1}
I20260928 15:55:40.962208 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:40.954336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:41051 (local address 127.22.157.193:36563)
0928 15:55:40.956004 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:40.956019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:40.956205 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:40.956527 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:40.956532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:40.956555 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:40.956997 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.957006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.957995 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.958003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.959733 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:40.959745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:40.960972 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:40.960978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:40.961147 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:40.961169 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 15:55:40.961269 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 15:55:40.961394 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1566,"mutex_wait_us":117,"spinlock_wait_cycles":18176,"thread_start_us":1068,"threads_started":1}
I20260928 15:55:40.963019 9959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4e393246db4ce19b1270dd3ef4fb43" candidate_uuid: "c471e68546434550a8a679797fe16e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" is_pre_election: true
I20260928 15:55:40.963240 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4e393246db4ce19b1270dd3ef4fb43" candidate_uuid: "c471e68546434550a8a679797fe16e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230b025f0a74eddaaa31a4c3d895629" is_pre_election: true
I20260928 15:55:40.963660 9959 raft_consensus.cc:2454] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c471e68546434550a8a679797fe16e9b in term 0.
I20260928 15:55:40.963723 10089 raft_consensus.cc:2454] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c471e68546434550a8a679797fe16e9b in term 0.
I20260928 15:55:40.964409 9761 leader_election.cc:302] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c471e68546434550a8a679797fe16e9b, e230b025f0a74eddaaa31a4c3d895629; no voters:
I20260928 15:55:40.964792 10158 raft_consensus.cc:2781] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:55:40.964934 10158 raft_consensus.cc:482] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:55:40.965039 10158 raft_consensus.cc:3037] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:40.967192 10158 raft_consensus.cc:504] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } }
I20260928 15:55:40.967844 10158 leader_election.cc:288] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [CANDIDATE]: Term 1 election: Requested vote from peers d0c2d9c4cb5b4389a478de6ce7ce53e3 (127.22.157.194:41845), e230b025f0a74eddaaa31a4c3d895629 (127.22.157.195:41051)
I20260928 15:55:40.968308 9959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4e393246db4ce19b1270dd3ef4fb43" candidate_uuid: "c471e68546434550a8a679797fe16e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3"
I20260928 15:55:40.968369 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4e393246db4ce19b1270dd3ef4fb43" candidate_uuid: "c471e68546434550a8a679797fe16e9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e230b025f0a74eddaaa31a4c3d895629"
I20260928 15:55:40.968544 9959 raft_consensus.cc:3037] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:40.968602 10089 raft_consensus.cc:3037] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:40.971921 10089 raft_consensus.cc:2454] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c471e68546434550a8a679797fe16e9b in term 1.
I20260928 15:55:40.971930 9959 raft_consensus.cc:2454] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c471e68546434550a8a679797fe16e9b in term 1.
I20260928 15:55:40.972386 9761 leader_election.cc:302] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c471e68546434550a8a679797fe16e9b, e230b025f0a74eddaaa31a4c3d895629; no voters:
I20260928 15:55:40.972707 10158 raft_consensus.cc:2781] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:40.973191 10158 raft_consensus.cc:686] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 1 LEADER]: Becoming Leader. State: Replica: c471e68546434550a8a679797fe16e9b, State: Running, Role: LEADER
I20260928 15:55:40.973654 10158 consensus_queue.cc:237] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } }
I20260928 15:55:40.978266 9686 catalog_manager.cc:5495] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b reported cstate change: term changed from 0 to 1, leader changed from <none> to c471e68546434550a8a679797fe16e9b (127.22.157.193). New cstate: current_term: 1 leader_uuid: "c471e68546434550a8a679797fe16e9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c471e68546434550a8a679797fe16e9b" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 37185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 } health_report { overall_health: UNKNOWN } } }
W20260928 15:55:40.997505 10135 tablet.cc:2369] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:55:41.005679 10005 tablet.cc:2369] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:55:41.021870 9874 tablet.cc:2369] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:55:41.030838 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.025323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.22.157.193:37185)
0928 15:55:41.025546 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:41.025551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:41.025574 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:55:41.025702 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:41.025706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:41.025811 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:41.025959 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:55:41.026378 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:41.027130 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:41.028478 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:41.028983 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:41.029087 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:41.029607 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:55:41.030134 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:55:41.030270 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:41.030479 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 15:55:41.030527 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:55:41.040163 9959 raft_consensus.cc:1261] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [term 1 FOLLOWER]: Refusing update from remote peer c471e68546434550a8a679797fe16e9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:41.040163 10089 raft_consensus.cc:1261] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [term 1 FOLLOWER]: Refusing update from remote peer c471e68546434550a8a679797fe16e9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:41.041316 10163 consensus_queue.cc:1035] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e230b025f0a74eddaaa31a4c3d895629" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 41051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:41.041734 10158 consensus_queue.cc:1035] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c2d9c4cb5b4389a478de6ce7ce53e3" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 41845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:41.054912 10167 mvcc.cc:205] Tried to move back new op lower bound from 7334342414488317952 to 7334342414232203264. Current Snapshot: MvccSnapshot[applied={T|T < 7334342414488317952}]
I20260928 15:55:41.066383 9685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:41.066681 9685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:41.070430 9685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff7443faf6a145e888a37d229d9dcbad: Sending DeleteTablet for 3 replicas of tablet 9c4e393246db4ce19b1270dd3ef4fb43
I20260928 15:55:41.071719 9939 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c4e393246db4ce19b1270dd3ef4fb43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:41 UTC) from {username='slave'} at 127.0.0.1:40350
I20260928 15:55:41.071749 9807 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c4e393246db4ce19b1270dd3ef4fb43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:41 UTC) from {username='slave'} at 127.0.0.1:49660
I20260928 15:55:41.072075 10069 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c4e393246db4ce19b1270dd3ef4fb43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:41 UTC) from {username='slave'} at 127.0.0.1:46292
I20260928 15:55:41.075291 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 9745
I20260928 15:55:41.081776 10178 tablet_replica.cc:330] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: stopping tablet replica
I20260928 15:55:41.082894 10178 raft_consensus.cc:2229] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:41.083930 10176 tablet_replica.cc:330] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: stopping tablet replica
I20260928 15:55:41.084566 10178 raft_consensus.cc:2258] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:41.085116 10176 raft_consensus.cc:2229] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:55:41.086500 10176 raft_consensus.cc:2258] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:41.090317 10176 ts_tablet_manager.cc:1905] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:41.090535 10178 ts_tablet_manager.cc:1905] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:41.100229 10176 ts_tablet_manager.cc:1918] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:41.102456 10177 tablet_replica.cc:330] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: stopping tablet replica
I20260928 15:55:41.102869 10177 raft_consensus.cc:2229] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:41.103356 10177 raft_consensus.cc:2258] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:41.101244 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.076557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.22.157.193:37185)
0928 15:55:41.076822 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:41.076832 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:41.089690 (+ 12858us) server_negotiation.cc:407] Connection header received
0928 15:55:41.090292 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:41.090306 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:41.090438 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:41.090530 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:41.091429 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:41.093068 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:41.095717 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:41.096677 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:41.096856 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:41.098823 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:41.098838 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:41.098866 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:41.099981 (+ 1115us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:41.100070 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:41.100102 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:41.100213 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:41.100524 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 15:55:41.100686 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 15:55:41.105368 10177 ts_tablet_manager.cc:1905] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:41.105432 10176 log.cc:1196] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/9c4e393246db4ce19b1270dd3ef4fb43
I20260928 15:55:41.105628 9868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:41.106426 10178 ts_tablet_manager.cc:1918] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:41.106637 10178 log.cc:1196] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/9c4e393246db4ce19b1270dd3ef4fb43
I20260928 15:55:41.107272 10178 ts_tablet_manager.cc:1939] T 9c4e393246db4ce19b1270dd3ef4fb43 P e230b025f0a74eddaaa31a4c3d895629: Deleting consensus metadata
I20260928 15:55:41.109701 9671 catalog_manager.cc:4841] TS e230b025f0a74eddaaa31a4c3d895629 (127.22.157.195:41051): tablet 9c4e393246db4ce19b1270dd3ef4fb43 (table test-table [id=08b064b58896444b956c150563cbe6c3]) successfully deleted
I20260928 15:55:41.110610 10177 ts_tablet_manager.cc:1918] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:41.110780 10177 log.cc:1196] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/9c4e393246db4ce19b1270dd3ef4fb43
I20260928 15:55:41.111231 10177 ts_tablet_manager.cc:1939] T 9c4e393246db4ce19b1270dd3ef4fb43 P d0c2d9c4cb5b4389a478de6ce7ce53e3: Deleting consensus metadata
I20260928 15:55:41.113261 9670 catalog_manager.cc:4841] TS d0c2d9c4cb5b4389a478de6ce7ce53e3 (127.22.157.194:41845): tablet 9c4e393246db4ce19b1270dd3ef4fb43 (table test-table [id=08b064b58896444b956c150563cbe6c3]) successfully deleted
I20260928 15:55:41.211416 10176 ts_tablet_manager.cc:1939] T 9c4e393246db4ce19b1270dd3ef4fb43 P c471e68546434550a8a679797fe16e9b: Deleting consensus metadata
I20260928 15:55:41.212288 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 9745
I20260928 15:55:41.232825 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 9877
W20260928 15:55:41.233032 9673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.157.193:37185 (error 108)
I20260928 15:55:41.234943 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.234519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37185 (local address 127.0.0.1:49680)
0928 15:55:41.234693 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:41.234810 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 15:55:41.235179 9673 catalog_manager.cc:4584] TS c471e68546434550a8a679797fe16e9b (127.22.157.193:37185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c4e393246db4ce19b1270dd3ef4fb43: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37185: connect: Connection refused (error 111)
I20260928 15:55:41.240188 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.233689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.22.157.194:41845)
0928 15:55:41.233871 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:41.233877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:41.233981 (+ 104us) server_negotiation.cc:407] Connection header received
0928 15:55:41.234215 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:41.234221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:41.234335 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:41.234452 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:41.234894 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:41.235807 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:41.237182 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:41.237605 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:41.237760 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:41.238798 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:41.238807 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:41.238822 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:41.239406 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:41.239456 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:41.239476 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:41.239601 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:41.239745 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 15:55:41.239854 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:55:41.240993 9999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:41.289454 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.288861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37185 (local address 127.0.0.1:49682)
0928 15:55:41.289138 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:41.289255 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 15:55:41.290627 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.290216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37185 (local address 127.0.0.1:49698)
0928 15:55:41.290382 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:41.290470 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 15:55:41.332948 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 9877
I20260928 15:55:41.351380 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.350750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37185 (local address 127.0.0.1:49708)
0928 15:55:41.351017 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:41.351150 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:55:41.351970 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 10007
I20260928 15:55:41.352676 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.352231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37185 (local address 127.0.0.1:49724)
0928 15:55:41.352423 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:41.352492 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 15:55:41.359249 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.352849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.22.157.195:41051)
0928 15:55:41.353040 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:41.353044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:41.353063 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:55:41.353223 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:41.353227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:41.353317 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:41.353410 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:41.353771 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:41.354602 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:41.356052 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:41.356582 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:41.356689 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:41.357751 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:41.357760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:41.357775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:41.358339 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:41.358386 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:41.358405 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:41.358568 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:41.358831 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 15:55:41.358983 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:55:41.359889 10129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:41.433502 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.432922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37185 (local address 127.0.0.1:49736)
0928 15:55:41.433171 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:41.433286 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:55:41.434767 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:41.434300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:37185 (local address 127.0.0.1:49744)
0928 15:55:41.434511 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:41.434636 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:37185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 15:55:41.451197 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 10007
I20260928 15:55:41.468495 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 9655
I20260928 15:55:41.469712 9716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:41.581773 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 9655
2026-09-28T15:55:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2947 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:55:41 dist-test-slave-1vw0 krb5kdc[10187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:55:41 dist-test-slave-1vw0 krb5kdc[10187](info): set up 2 sockets
Sep 28 15:55:41 dist-test-slave-1vw0 krb5kdc[10187](info): commencing operation
krb5kdc: starting...
W20260928 15:55:42.721452 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:55:42 dist-test-slave-1vw0 krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:55:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:55:43.032866 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43685
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:43685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:43.381439 10200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:43.381824 10200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:43.382012 10200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:43.389511 10200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:43.389669 10200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:43.389780 10200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:43.389886 10200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:43.389961 10200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:43.400472 10200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:43685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:43.402117 10200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:43.403587 10200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:43.416988 10208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:43.417001 10206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:43.417001 10205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:43.417773 10200 server_base.cc:1034] running on GCE node
I20260928 15:55:43.418655 10200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:43.420158 10200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:43.421423 10200 hybrid_clock.cc:648] HybridClock initialized: now 1790610943421366 us; error 42 us; skew 500 ppm
I20260928 15:55:43.421860 10200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:43 dist-test-slave-1vw0 krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610943, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:43.425160 10200 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:55:43.426626 10200 webserver.cc:460] Webserver started at http://127.22.157.254:42749/ using document root <none> and password file <none>
I20260928 15:55:43.427279 10200 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:43.427386 10200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:43.427654 10200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:43.430948 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "886b8a7cb1d54a20901281781060a7fe"
format_stamp: "Formatted at 2026-09-28 15:55:43 on dist-test-slave-1vw0"
server_key: "cc5cb7e36b78ab9ebfeb7a358692f2e3"
server_key_iv: "aa1b044e2e6fa4409996dd070fa8130c"
server_key_version: "encryptionkey@0"
I20260928 15:55:43.431734 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "886b8a7cb1d54a20901281781060a7fe"
format_stamp: "Formatted at 2026-09-28 15:55:43 on dist-test-slave-1vw0"
server_key: "cc5cb7e36b78ab9ebfeb7a358692f2e3"
server_key_iv: "aa1b044e2e6fa4409996dd070fa8130c"
server_key_version: "encryptionkey@0"
I20260928 15:55:43.437314 10200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:55:43.441107 10215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:43.442667 10200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:55:43.442821 10200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "886b8a7cb1d54a20901281781060a7fe"
format_stamp: "Formatted at 2026-09-28 15:55:43 on dist-test-slave-1vw0"
server_key: "cc5cb7e36b78ab9ebfeb7a358692f2e3"
server_key_iv: "aa1b044e2e6fa4409996dd070fa8130c"
server_key_version: "encryptionkey@0"
I20260928 15:55:43.443001 10200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:43.466413 10200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:43.471319 10200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:43.471698 10200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:43.490669 10200 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:43685
I20260928 15:55:43.490762 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:43685 every 8 connection(s)
I20260928 15:55:43.492182 10200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:43.496377 10267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:43.502316 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 10200
I20260928 15:55:43.502525 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:43.502980 23159 external_mini_cluster.cc:1428] Setting key e6769dc9415281b495c1501facb8d8c9
I20260928 15:55:43.508919 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe: Bootstrap starting.
I20260928 15:55:43.512800 10267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:43.513990 10267 log.cc:824] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:43.517102 10267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe: No bootstrap required, opened a new log
Sep 28 15:55:43 dist-test-slave-1vw0 krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:43.528362 10267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886b8a7cb1d54a20901281781060a7fe" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43685 } }
I20260928 15:55:43.528934 10267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:43.529080 10267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886b8a7cb1d54a20901281781060a7fe, State: Initialized, Role: FOLLOWER
I20260928 15:55:43.529841 10267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886b8a7cb1d54a20901281781060a7fe" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43685 } }
I20260928 15:55:43.530128 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:43.505026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.22.157.254:43685)
0928 15:55:43.505918 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:43.505938 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:43.505999 (+ 61us) server_negotiation.cc:407] Connection header received
0928 15:55:43.508429 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:43.508461 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:43.509059 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:43.509528 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:43.510918 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:43.512925 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:43.514406 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:43.515337 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:43.515536 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:43.519916 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:43.519926 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:43.519960 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:43.520008 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:43.524260 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:43.525160 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:43.525169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:43.525178 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:43.525531 (+ 353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:43.526122 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:43.526127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:43.526131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:43.527152 (+ 1021us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:43.527481 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:43.527964 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 15:55:43.528483 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":362,"threads_started":1}
I20260928 15:55:43.530179 10267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:43.530313 10267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:43.530507 10267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:43.533681 10267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886b8a7cb1d54a20901281781060a7fe" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43685 } }
I20260928 15:55:43.534240 10267 leader_election.cc:302] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 886b8a7cb1d54a20901281781060a7fe; no voters:
I20260928 15:55:43.534888 10267 leader_election.cc:288] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:43.535027 10272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:43.535802 10272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [term 1 LEADER]: Becoming Leader. State: Replica: 886b8a7cb1d54a20901281781060a7fe, State: Running, Role: LEADER
I20260928 15:55:43.536358 10272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886b8a7cb1d54a20901281781060a7fe" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43685 } }
I20260928 15:55:43.537165 10267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:43.539198 10274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886b8a7cb1d54a20901281781060a7fe. Latest consensus state: current_term: 1 leader_uuid: "886b8a7cb1d54a20901281781060a7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886b8a7cb1d54a20901281781060a7fe" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43685 } } }
I20260928 15:55:43.539609 10274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:43.540396 10273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "886b8a7cb1d54a20901281781060a7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886b8a7cb1d54a20901281781060a7fe" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43685 } } }
I20260928 15:55:43.540799 10273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:43.542440 10277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:43.548012 10277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:43.559265 10277 catalog_manager.cc:1347] Generated new cluster ID: 12ee60283c83451ab00ae7dc770c1dbc
I20260928 15:55:43.559392 10277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:43.573215 10277 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:43.574636 10277 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:43.585690 10277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe: Generated new TSK 0
I20260928 15:55:43.586556 10277 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
Entry for principal kudu/127.22.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab.
I20260928 15:55:43.660328 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43685
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:44.018038 10293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:44.018332 10293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:44.018429 10293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:44.025969 10293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:44.026113 10293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:44.026234 10293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:55:44.037634 10293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.193.keytab
--principal=kudu/127.22.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:44.039163 10293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:44.040490 10293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:44.054610 10298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:44.054559 10299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:44.054528 10301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:44.055465 10293 server_base.cc:1034] running on GCE node
I20260928 15:55:44.056135 10293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:44.057334 10293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:44.058583 10293 hybrid_clock.cc:648] HybridClock initialized: now 1790610944058462 us; error 85 us; skew 500 ppm
I20260928 15:55:44.059141 10293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:44 dist-test-slave-1vw0 krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610944, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:44.062742 10293 init.cc:377] Logged in from keytab as kudu/127.22.157.193@KRBTEST.COM (short username kudu)
I20260928 15:55:44.064247 10293 webserver.cc:460] Webserver started at http://127.22.157.193:36031/ using document root <none> and password file <none>
I20260928 15:55:44.065052 10293 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:44.065201 10293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:44.065524 10293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:44.068594 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "aaad221777f44dc1a92bc116b7ff4ce0"
format_stamp: "Formatted at 2026-09-28 15:55:44 on dist-test-slave-1vw0"
server_key: "8c81c722c44a024f2e87957c2580c726"
server_key_iv: "2ca4589dcec70e6ac1c062a6797c8161"
server_key_version: "encryptionkey@0"
I20260928 15:55:44.069572 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "aaad221777f44dc1a92bc116b7ff4ce0"
format_stamp: "Formatted at 2026-09-28 15:55:44 on dist-test-slave-1vw0"
server_key: "8c81c722c44a024f2e87957c2580c726"
server_key_iv: "2ca4589dcec70e6ac1c062a6797c8161"
server_key_version: "encryptionkey@0"
I20260928 15:55:44.075111 10293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:55:44.078701 10308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:44.080168 10293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 15:55:44.080391 10293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "aaad221777f44dc1a92bc116b7ff4ce0"
format_stamp: "Formatted at 2026-09-28 15:55:44 on dist-test-slave-1vw0"
server_key: "8c81c722c44a024f2e87957c2580c726"
server_key_iv: "2ca4589dcec70e6ac1c062a6797c8161"
server_key_version: "encryptionkey@0"
I20260928 15:55:44.080695 10293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:44.095204 10293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:44.099327 10293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:44.099658 10293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:44.100649 10293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:44.102044 10293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:44.102147 10293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:44.102298 10293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:44.102380 10293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:44.137319 10293 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:33015
I20260928 15:55:44.137409 10420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:33015 every 8 connection(s)
I20260928 15:55:44.138715 10293 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:55:44.139616 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 10293
I20260928 15:55:44.139817 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:55:44.140372 23159 external_mini_cluster.cc:1428] Setting key a6abed08ee60286504adbf560faaed0c
Sep 28 15:55:44 dist-test-slave-1vw0 krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610944, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.193@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:44.162243 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:44.141774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:56097 (local address 127.22.157.254:43685)
0928 15:55:44.142325 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:44.142332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:44.143501 (+ 1169us) server_negotiation.cc:407] Connection header received
0928 15:55:44.145475 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:44.145484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:44.145578 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:44.145702 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:44.147429 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:44.148601 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:44.150244 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:44.150986 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:44.151131 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:44.155467 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:44.155473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:44.155476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:44.155516 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:44.158658 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:44.159633 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:44.159639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:44.159641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:44.159777 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:44.160281 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:44.160286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:44.160288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:44.160578 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:44.160789 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:44.161594 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 15:55:44.161854 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":314,"threads_started":1}
I20260928 15:55:44.163313 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:44.142474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43685 (local address 127.22.157.193:56097)
0928 15:55:44.143320 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:44.143370 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:44.145218 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:44.145914 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:44.145923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:44.146567 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:44.147261 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:44.147280 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:44.148804 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:44.148808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:44.150068 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:44.150081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:44.151288 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:44.151293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:44.151502 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:44.152227 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:55:44.152250 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:44.155164 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:44.158891 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:44.158902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:44.158928 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:44.159464 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:44.159901 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:44.159906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:44.159910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:44.160142 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:44.160751 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:44.160761 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:44.161175 (+ 414us) client_negotiation.cc:769] Sending connection context
0928 15:55:44.161462 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 15:55:44.162093 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":350,"threads_started":1}
I20260928 15:55:44.165311 10421 heartbeater.cc:347] Connected to a master server at 127.22.157.254:43685
I20260928 15:55:44.165704 10421 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:44.166579 10421 heartbeater.cc:511] Master 127.22.157.254:43685 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.157.194@KRBTEST.COM; defaulting to no policy
I20260928 15:55:44.169157 10232 ts_manager.cc:194] Registered new tserver with Master: aaad221777f44dc1a92bc116b7ff4ce0 (127.22.157.193:33015)
I20260928 15:55:44.170763 10232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.193@KRBTEST.COM'} at 127.22.157.193:56097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
Entry for principal kudu/127.22.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab.
I20260928 15:55:44.205121 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43685
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:44.553069 10427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:44.553354 10427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:44.553519 10427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:44.561112 10427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:44.561241 10427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:44.561452 10427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:55:44.571489 10427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.194.keytab
--principal=kudu/127.22.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:44.572999 10427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:44.574388 10427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:44.587920 10435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:44.588168 10432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:44.588207 10433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:44.589176 10427 server_base.cc:1034] running on GCE node
I20260928 15:55:44.589856 10427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:44.590942 10427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:44.592178 10427 hybrid_clock.cc:648] HybridClock initialized: now 1790610944592120 us; error 52 us; skew 500 ppm
I20260928 15:55:44.592517 10427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:44 dist-test-slave-1vw0 krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610944, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:44.595788 10427 init.cc:377] Logged in from keytab as kudu/127.22.157.194@KRBTEST.COM (short username kudu)
I20260928 15:55:44.597398 10427 webserver.cc:460] Webserver started at http://127.22.157.194:41501/ using document root <none> and password file <none>
I20260928 15:55:44.598100 10427 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:44.598248 10427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:44.598563 10427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:44.601802 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "3bae8df63cb94ef3b99a6525ce6a046d"
format_stamp: "Formatted at 2026-09-28 15:55:44 on dist-test-slave-1vw0"
server_key: "bc2cdd90a703bca70691d9547a523ace"
server_key_iv: "08499cc8e7f3173d3587d5693aabee9c"
server_key_version: "encryptionkey@0"
I20260928 15:55:44.602573 10427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "3bae8df63cb94ef3b99a6525ce6a046d"
format_stamp: "Formatted at 2026-09-28 15:55:44 on dist-test-slave-1vw0"
server_key: "bc2cdd90a703bca70691d9547a523ace"
server_key_iv: "08499cc8e7f3173d3587d5693aabee9c"
server_key_version: "encryptionkey@0"
I20260928 15:55:44.607817 10427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:55:44.611445 10442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:44.612955 10427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:44.613137 10427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "3bae8df63cb94ef3b99a6525ce6a046d"
format_stamp: "Formatted at 2026-09-28 15:55:44 on dist-test-slave-1vw0"
server_key: "bc2cdd90a703bca70691d9547a523ace"
server_key_iv: "08499cc8e7f3173d3587d5693aabee9c"
server_key_version: "encryptionkey@0"
I20260928 15:55:44.613359 10427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:44.630105 10427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:44.634776 10427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:44.635100 10427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:44.636173 10427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:44.637866 10427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:44.637969 10427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:44.638139 10427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:44.638211 10427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:44.673771 10427 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:44075
I20260928 15:55:44.673822 10554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:44075 every 8 connection(s)
I20260928 15:55:44.675204 10427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:55:44.684703 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 10427
I20260928 15:55:44.684938 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:55:44.685501 23159 external_mini_cluster.cc:1428] Setting key 9606f7ba8d29968d2cbbf37e507810e4
Sep 28 15:55:44 dist-test-slave-1vw0 krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610944, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.194@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:44.696640 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:44.677978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:48003 (local address 127.22.157.254:43685)
0928 15:55:44.678447 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:44.678451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:44.679306 (+ 855us) server_negotiation.cc:407] Connection header received
0928 15:55:44.681256 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:44.681262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:44.681357 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:44.681476 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:44.683033 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:44.683774 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:44.686042 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:44.686749 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:44.686880 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:44.690641 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:44.690647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:44.690650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:44.690685 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:44.693371 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:44.694180 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:44.694185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:44.694187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:44.694324 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:44.694802 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:44.694811 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:44.694813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:44.695090 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:44.695251 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:44.695989 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 15:55:44.696232 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":265,"threads_started":1}
I20260928 15:55:44.697732 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:44.678449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43685 (local address 127.22.157.194:48003)
0928 15:55:44.679151 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:44.679187 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:44.680967 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:44.681738 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:44.681746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:44.682285 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:44.682864 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:44.682881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:44.683937 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:44.683941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:44.685893 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:44.685908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:44.686957 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:44.686962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:44.687197 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:44.687878 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:55:44.687902 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:44.690379 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:44.693528 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:44.693537 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:44.693556 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:44.694013 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:44.694446 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:44.694451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:44.694455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:44.694678 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:44.695223 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:44.695232 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:44.695620 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 15:55:44.695901 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 15:55:44.696494 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":271,"threads_started":1}
I20260928 15:55:44.699304 10555 heartbeater.cc:347] Connected to a master server at 127.22.157.254:43685
I20260928 15:55:44.699635 10555 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:44.700367 10555 heartbeater.cc:511] Master 127.22.157.254:43685 requested a full tablet report, sending...
I20260928 15:55:44.701916 10232 ts_manager.cc:194] Registered new tserver with Master: 3bae8df63cb94ef3b99a6525ce6a046d (127.22.157.194:44075)
I20260928 15:55:44.702750 10232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.194@KRBTEST.COM'} at 127.22.157.194:48003
WARNING: no policy specified for kudu/127.22.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
Entry for principal kudu/127.22.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab.
I20260928 15:55:44.746440 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43685
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:37781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:45.089448 10561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:45.089752 10561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:45.089846 10561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:45.097198 10561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:45.097324 10561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:45.097438 10561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:55:45.107295 10561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:37781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.195.keytab
--principal=kudu/127.22.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:43685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:45.108590 10561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:45.109953 10561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:45.123739 10569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:45.123975 10566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:45.123734 10567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:45.125099 10561 server_base.cc:1034] running on GCE node
I20260928 15:55:45.125778 10561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:45.126801 10561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:45.128007 10561 hybrid_clock.cc:648] HybridClock initialized: now 1790610945127967 us; error 27 us; skew 500 ppm
I20260928 15:55:45.128401 10561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:45 dist-test-slave-1vw0 krb5kdc[10187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610945, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:45.131650 10561 init.cc:377] Logged in from keytab as kudu/127.22.157.195@KRBTEST.COM (short username kudu)
I20260928 15:55:45.133258 10561 webserver.cc:460] Webserver started at http://127.22.157.195:43059/ using document root <none> and password file <none>
I20260928 15:55:45.134019 10561 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:45.134182 10561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:45.134508 10561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:45.137840 10561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "22beb3125f97445e945991427a503db2"
format_stamp: "Formatted at 2026-09-28 15:55:45 on dist-test-slave-1vw0"
server_key: "33cb68f62d26bca6edd7ce6eb9130d1b"
server_key_iv: "181c2988d7eb2bddf3dda375872cb95d"
server_key_version: "encryptionkey@0"
I20260928 15:55:45.138751 10561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "22beb3125f97445e945991427a503db2"
format_stamp: "Formatted at 2026-09-28 15:55:45 on dist-test-slave-1vw0"
server_key: "33cb68f62d26bca6edd7ce6eb9130d1b"
server_key_iv: "181c2988d7eb2bddf3dda375872cb95d"
server_key_version: "encryptionkey@0"
I20260928 15:55:45.143946 10561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:55:45.147603 10576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:45.149034 10561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:45.149205 10561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "22beb3125f97445e945991427a503db2"
format_stamp: "Formatted at 2026-09-28 15:55:45 on dist-test-slave-1vw0"
server_key: "33cb68f62d26bca6edd7ce6eb9130d1b"
server_key_iv: "181c2988d7eb2bddf3dda375872cb95d"
server_key_version: "encryptionkey@0"
I20260928 15:55:45.149398 10561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:45.171149 10561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:45.174517 10421 heartbeater.cc:503] Master 127.22.157.254:43685 was elected leader, sending a full tablet report...
I20260928 15:55:45.175591 10561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:45.175886 10561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:45.177233 10561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:45.178589 10561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:45.178668 10561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:45.178841 10561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:45.178891 10561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:45.212837 10561 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:39243
I20260928 15:55:45.212893 10688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:39243 every 8 connection(s)
I20260928 15:55:45.214219 10561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:55:45.215173 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 10561
I20260928 15:55:45.215361 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:55:45.215893 23159 external_mini_cluster.cc:1428] Setting key 19e142dc070c968cc7fde44493392731
Sep 28 15:55:45 dist-test-slave-1vw0 krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610945, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.195@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:55:45.235286 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.217108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:54945 (local address 127.22.157.254:43685)
0928 15:55:45.217567 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.217573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.218921 (+ 1348us) server_negotiation.cc:407] Connection header received
0928 15:55:45.220857 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.220861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.220962 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.221088 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:45.222572 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.223349 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.224974 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.225381 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.225514 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.229156 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:45.229162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:45.229167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:45.229221 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:45.232383 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.233199 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.233203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.233205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.233326 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.233711 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.233714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.233716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.233991 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:45.234117 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.234835 (+ 718us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.235013 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":279,"threads_started":1}
I20260928 15:55:45.236265 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.217773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43685 (local address 127.22.157.195:54945)
0928 15:55:45.218767 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:45.218821 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:45.220553 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:45.221325 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:45.221333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:45.221837 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:45.222401 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.222419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.223507 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.223515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.224826 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.224838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.225617 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.225624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.225866 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.226609 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:55:45.226649 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:45.228932 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:45.232515 (+ 3583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:45.232522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:45.232537 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:45.233078 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:45.233416 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:55:45.233420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:55:45.233422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:55:45.233600 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:55:45.234100 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:45.234106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:45.234445 (+ 339us) client_negotiation.cc:769] Sending connection context
0928 15:55:45.234680 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 15:55:45.235085 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":442,"threads_started":1}
I20260928 15:55:45.237821 10689 heartbeater.cc:347] Connected to a master server at 127.22.157.254:43685
I20260928 15:55:45.238145 10689 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:45.238879 10689 heartbeater.cc:511] Master 127.22.157.254:43685 requested a full tablet report, sending...
I20260928 15:55:45.240303 10232 ts_manager.cc:194] Registered new tserver with Master: 22beb3125f97445e945991427a503db2 (127.22.157.195:39243)
I20260928 15:55:45.241171 10232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.157.195@KRBTEST.COM'} at 127.22.157.195:54945
I20260928 15:55:45.244475 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:55:45.262598 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.249200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.22.157.254:43685)
0928 15:55:45.249381 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.249386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.249906 (+ 520us) server_negotiation.cc:407] Connection header received
0928 15:55:45.250017 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.250021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.250112 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.250218 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:45.251658 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.252413 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.253944 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.254477 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.254606 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.257160 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:45.257165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:45.257168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:45.257205 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:45.259916 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.260624 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.260629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.260631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.260850 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.261233 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.261237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.261239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.261623 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:45.261820 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.262129 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.262327 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:55:45.266774 10232 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:55:45.273860 10232 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:55:45.312186 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.288352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.22.157.195:39243)
0928 15:55:45.303076 (+ 14724us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.303088 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.303125 (+ 37us) server_negotiation.cc:407] Connection header received
0928 15:55:45.303245 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.303253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.303542 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.304770 (+ 1228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:45.305091 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.306756 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.309397 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.310765 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.311323 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.311504 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.311627 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.311825 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14539,"thread_start_us":14771,"threads_started":1}
I20260928 15:55:45.312453 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.286759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:44075 (local address 127.0.0.1:54554)
0928 15:55:45.296421 (+ 9662us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:45.296473 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:45.296662 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:45.304580 (+ 7918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:45.304587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:45.304642 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:45.305135 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.305149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.307206 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.307211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.309019 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.309036 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.310970 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.310984 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.311301 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.311452 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 15:55:45.311587 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 15:55:45.311858 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9386,"mutex_wait_us":189,"thread_start_us":16198,"threads_started":1}
I20260928 15:55:45.312474 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.287637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:39243 (local address 127.0.0.1:47782)
0928 15:55:45.297164 (+ 9527us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:45.297185 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:45.297314 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:45.303833 (+ 6519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:45.303840 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:45.303899 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:45.304547 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.304563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.306895 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.306900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.309221 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.309235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.310996 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.311000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.311358 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.311452 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 15:55:45.311588 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 15:55:45.311858 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9352,"mutex_wait_us":183,"spinlock_wait_cycles":3456,"thread_start_us":700,"threads_started":1}
I20260928 15:55:45.313030 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.286942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.22.157.194:44075)
0928 15:55:45.303986 (+ 17044us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.303995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.304022 (+ 27us) server_negotiation.cc:407] Connection header received
0928 15:55:45.304125 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.304133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.304353 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.305365 (+ 1012us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:45.305595 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.307060 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.309186 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.310734 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.310911 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.311101 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.312458 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.312622 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16886,"thread_start_us":16746,"threads_started":1}
I20260928 15:55:45.313055 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.289445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:33015 (local address 127.0.0.1:43700)
0928 15:55:45.302733 (+ 13288us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:45.302756 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:45.302879 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:45.303859 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:45.303863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:45.303916 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:45.304681 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.304697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.306495 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.306502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.309138 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.309162 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.311361 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.311367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.311561 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.311598 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 15:55:45.311675 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 15:55:45.312597 (+ 922us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13126,"mutex_wait_us":846,"spinlock_wait_cycles":2432,"thread_start_us":6534,"threads_started":1}
I20260928 15:55:45.313267 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.290120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43700 (local address 127.22.157.193:33015)
0928 15:55:45.303044 (+ 12924us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.303059 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.303103 (+ 44us) server_negotiation.cc:407] Connection header received
0928 15:55:45.303261 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.303273 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.303601 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.304074 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:45.305108 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.306338 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.309335 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.310730 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.310944 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.311101 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.311826 (+ 725us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.312000 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12663,"thread_start_us":13203,"threads_started":1}
I20260928 15:55:45.317279 10490 tablet_service.cc:1467] Processing CreateTablet for tablet 8b55b2f9005e4a72b8499a6e214f8017 (DEFAULT_TABLE table=test-table [id=add310a217fc43429bfdcf9b6b3836ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:45.318092 10624 tablet_service.cc:1467] Processing CreateTablet for tablet 8b55b2f9005e4a72b8499a6e214f8017 (DEFAULT_TABLE table=test-table [id=add310a217fc43429bfdcf9b6b3836ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:45.318804 10490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b55b2f9005e4a72b8499a6e214f8017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:45.319346 10624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b55b2f9005e4a72b8499a6e214f8017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:45.326332 10356 tablet_service.cc:1467] Processing CreateTablet for tablet 8b55b2f9005e4a72b8499a6e214f8017 (DEFAULT_TABLE table=test-table [id=add310a217fc43429bfdcf9b6b3836ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:45.327939 10356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b55b2f9005e4a72b8499a6e214f8017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:45.330327 10708 tablet_bootstrap.cc:492] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: Bootstrap starting.
I20260928 15:55:45.333969 10708 tablet_bootstrap.cc:654] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:45.335028 10708 log.cc:824] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:45.338076 10708 tablet_bootstrap.cc:492] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: No bootstrap required, opened a new log
I20260928 15:55:45.338385 10708 ts_tablet_manager.cc:1397] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 15:55:45.342639 10711 tablet_bootstrap.cc:492] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: Bootstrap starting.
I20260928 15:55:45.343218 10709 tablet_bootstrap.cc:492] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: Bootstrap starting.
I20260928 15:55:45.346388 10708 raft_consensus.cc:348] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } }
I20260928 15:55:45.346863 10708 raft_consensus.cc:374] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:45.347009 10708 raft_consensus.cc:729] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bae8df63cb94ef3b99a6525ce6a046d, State: Initialized, Role: FOLLOWER
I20260928 15:55:45.347149 10709 tablet_bootstrap.cc:654] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:45.347149 10711 tablet_bootstrap.cc:654] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:45.347729 10708 consensus_queue.cc:260] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } }
I20260928 15:55:45.348680 10711 log.cc:824] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:45.349009 10555 heartbeater.cc:503] Master 127.22.157.254:43685 was elected leader, sending a full tablet report...
I20260928 15:55:45.348691 10709 log.cc:824] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:45.349726 10708 ts_tablet_manager.cc:1428] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 15:55:45.351751 10711 tablet_bootstrap.cc:492] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: No bootstrap required, opened a new log
I20260928 15:55:45.352101 10711 ts_tablet_manager.cc:1397] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 15:55:45.354943 10709 tablet_bootstrap.cc:492] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: No bootstrap required, opened a new log
I20260928 15:55:45.355247 10709 ts_tablet_manager.cc:1397] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.006s
I20260928 15:55:45.360615 10711 raft_consensus.cc:348] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } }
I20260928 15:55:45.361115 10711 raft_consensus.cc:374] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:45.361240 10711 raft_consensus.cc:729] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaad221777f44dc1a92bc116b7ff4ce0, State: Initialized, Role: FOLLOWER
I20260928 15:55:45.361837 10711 consensus_queue.cc:260] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } }
I20260928 15:55:45.363253 10709 raft_consensus.cc:348] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } }
I20260928 15:55:45.363616 10711 ts_tablet_manager.cc:1428] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260928 15:55:45.363610 10709 raft_consensus.cc:374] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:45.363746 10709 raft_consensus.cc:729] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22beb3125f97445e945991427a503db2, State: Initialized, Role: FOLLOWER
I20260928 15:55:45.364317 10709 consensus_queue.cc:260] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } }
I20260928 15:55:45.365513 10689 heartbeater.cc:503] Master 127.22.157.254:43685 was elected leader, sending a full tablet report...
I20260928 15:55:45.366115 10709 ts_tablet_manager.cc:1428] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
W20260928 15:55:45.391302 10422 tablet.cc:2369] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:55:45.427428 10556 tablet.cc:2369] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:55:45.465898 10690 tablet.cc:2369] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:55:45.480621 10715 raft_consensus.cc:482] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:55:45.480997 10715 raft_consensus.cc:504] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } }
I20260928 15:55:45.482856 10715 leader_election.cc:288] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bae8df63cb94ef3b99a6525ce6a046d (127.22.157.194:44075), 22beb3125f97445e945991427a503db2 (127.22.157.195:39243)
I20260928 15:55:45.490116 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.483605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:40093 (local address 127.22.157.195:39243)
0928 15:55:45.483853 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.483862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.484099 (+ 237us) server_negotiation.cc:407] Connection header received
0928 15:55:45.484481 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.484486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.484568 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.484683 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:45.485374 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.486262 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.488153 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.489098 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.489221 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.489274 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.489683 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.489831 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:55:45.490201 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.483184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:47019 (local address 127.22.157.194:44075)
0928 15:55:45.483413 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.483422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.484524 (+ 1102us) server_negotiation.cc:407] Connection header received
0928 15:55:45.484701 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.484707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.484807 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.484872 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:45.485753 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.486550 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.488408 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.489272 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.489428 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.489500 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.489816 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.489963 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:55:45.490809 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.483184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:39243 (local address 127.22.157.193:40093)
0928 15:55:45.483979 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:45.483999 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:45.484248 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:45.484754 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:45.484758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:45.484782 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:45.485209 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.485229 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.486393 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.486398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.487963 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.487975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.489312 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.489320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.489508 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.489555 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 15:55:45.489635 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 15:55:45.489754 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":464,"threads_started":1}
I20260928 15:55:45.490942 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.483307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:44075 (local address 127.22.157.193:47019)
0928 15:55:45.484357 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:45.484371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:45.484525 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:45.485121 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:45.485125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:45.485153 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:45.485575 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.485588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.486713 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.486717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.488251 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:45.488266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.489485 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.489489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:45.489638 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.489672 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 15:55:45.489764 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:55:45.489879 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":750,"threads_started":1}
I20260928 15:55:45.491850 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b55b2f9005e4a72b8499a6e214f8017" candidate_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22beb3125f97445e945991427a503db2" is_pre_election: true
I20260928 15:55:45.491932 10510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b55b2f9005e4a72b8499a6e214f8017" candidate_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" is_pre_election: true
I20260928 15:55:45.492389 10644 raft_consensus.cc:2454] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaad221777f44dc1a92bc116b7ff4ce0 in term 0.
I20260928 15:55:45.492424 10510 raft_consensus.cc:2454] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaad221777f44dc1a92bc116b7ff4ce0 in term 0.
I20260928 15:55:45.493139 10310 leader_election.cc:302] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22beb3125f97445e945991427a503db2, 3bae8df63cb94ef3b99a6525ce6a046d, aaad221777f44dc1a92bc116b7ff4ce0; no voters:
I20260928 15:55:45.493628 10715 raft_consensus.cc:2781] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:55:45.493801 10715 raft_consensus.cc:482] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:55:45.493907 10715 raft_consensus.cc:3037] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:45.497136 10715 raft_consensus.cc:504] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } }
I20260928 15:55:45.497864 10715 leader_election.cc:288] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [CANDIDATE]: Term 1 election: Requested vote from peers 3bae8df63cb94ef3b99a6525ce6a046d (127.22.157.194:44075), 22beb3125f97445e945991427a503db2 (127.22.157.195:39243)
I20260928 15:55:45.498353 10510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b55b2f9005e4a72b8499a6e214f8017" candidate_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bae8df63cb94ef3b99a6525ce6a046d"
I20260928 15:55:45.498370 10644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b55b2f9005e4a72b8499a6e214f8017" candidate_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22beb3125f97445e945991427a503db2"
I20260928 15:55:45.498575 10510 raft_consensus.cc:3037] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:45.498589 10644 raft_consensus.cc:3037] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:45.500815 10510 raft_consensus.cc:2454] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaad221777f44dc1a92bc116b7ff4ce0 in term 1.
I20260928 15:55:45.501212 10310 leader_election.cc:302] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bae8df63cb94ef3b99a6525ce6a046d, aaad221777f44dc1a92bc116b7ff4ce0; no voters:
I20260928 15:55:45.501525 10715 raft_consensus.cc:2781] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:45.501802 10644 raft_consensus.cc:2454] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaad221777f44dc1a92bc116b7ff4ce0 in term 1.
I20260928 15:55:45.502060 10715 raft_consensus.cc:686] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 1 LEADER]: Becoming Leader. State: Replica: aaad221777f44dc1a92bc116b7ff4ce0, State: Running, Role: LEADER
I20260928 15:55:45.502642 10715 consensus_queue.cc:237] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } }
I20260928 15:55:45.507208 10231 catalog_manager.cc:5495] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaad221777f44dc1a92bc116b7ff4ce0 (127.22.157.193). New cstate: current_term: 1 leader_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaad221777f44dc1a92bc116b7ff4ce0" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 33015 } health_report { overall_health: HEALTHY } } }
I20260928 15:55:45.543987 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.538325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.22.157.193:33015)
0928 15:55:45.538582 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.538589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.538611 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:55:45.538840 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.538844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.538944 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.539130 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:55:45.539528 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.540248 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.541768 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.542188 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.542325 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.542856 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:55:45.543395 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:55:45.543578 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.543628 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.543717 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 15:55:45.553030 10510 raft_consensus.cc:1261] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [term 1 FOLLOWER]: Refusing update from remote peer aaad221777f44dc1a92bc116b7ff4ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:45.553061 10644 raft_consensus.cc:1261] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [term 1 FOLLOWER]: Refusing update from remote peer aaad221777f44dc1a92bc116b7ff4ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:45.554126 10719 consensus_queue.cc:1035] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22beb3125f97445e945991427a503db2" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 39243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:45.554502 10715 consensus_queue.cc:1035] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bae8df63cb94ef3b99a6525ce6a046d" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:45.568234 10724 mvcc.cc:205] Tried to move back new op lower bound from 7334342432974372864 to 7334342432782802944. Current Snapshot: MvccSnapshot[applied={T|T < 7334342432974372864 or (T in {7334342432974372864})}]
I20260928 15:55:45.574918 10723 mvcc.cc:205] Tried to move back new op lower bound from 7334342432974372864 to 7334342432782802944. Current Snapshot: MvccSnapshot[applied={T|T < 7334342432974372864}]
I20260928 15:55:45.582022 10232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:45.582381 10232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:45.586737 10232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 886b8a7cb1d54a20901281781060a7fe: Sending DeleteTablet for 3 replicas of tablet 8b55b2f9005e4a72b8499a6e214f8017
I20260928 15:55:45.587952 10490 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b55b2f9005e4a72b8499a6e214f8017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:45 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:54554
I20260928 15:55:45.588279 10356 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b55b2f9005e4a72b8499a6e214f8017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:45 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:43700
I20260928 15:55:45.588300 10624 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b55b2f9005e4a72b8499a6e214f8017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:45 UTC) from {username='kudu', principal='kudu/127.22.157.254@KRBTEST.COM'} at 127.0.0.1:47782
I20260928 15:55:45.589249 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 10293
I20260928 15:55:45.600809 10732 tablet_replica.cc:330] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: stopping tablet replica
I20260928 15:55:45.601569 10732 raft_consensus.cc:2229] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:45.602381 10732 raft_consensus.cc:2258] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:45.605288 10732 ts_tablet_manager.cc:1905] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:45.607061 10734 tablet_replica.cc:330] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: stopping tablet replica
I20260928 15:55:45.607501 10734 raft_consensus.cc:2229] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:45.607960 10734 raft_consensus.cc:2258] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:45.608707 10733 tablet_replica.cc:330] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: stopping tablet replica
I20260928 15:55:45.609205 10733 raft_consensus.cc:2229] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:55:45.609953 10733 raft_consensus.cc:2258] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:45.610687 10734 ts_tablet_manager.cc:1905] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:45.612121 10733 ts_tablet_manager.cc:1905] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:45.612618 10732 ts_tablet_manager.cc:1918] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:45.612949 10732 log.cc:1196] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/8b55b2f9005e4a72b8499a6e214f8017
I20260928 15:55:45.613621 10732 ts_tablet_manager.cc:1939] T 8b55b2f9005e4a72b8499a6e214f8017 P 3bae8df63cb94ef3b99a6525ce6a046d: Deleting consensus metadata
I20260928 15:55:45.615772 10217 catalog_manager.cc:4841] TS 3bae8df63cb94ef3b99a6525ce6a046d (127.22.157.194:44075): tablet 8b55b2f9005e4a72b8499a6e214f8017 (table test-table [id=add310a217fc43429bfdcf9b6b3836ad]) successfully deleted
I20260928 15:55:45.618184 10734 ts_tablet_manager.cc:1918] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:45.618395 10734 log.cc:1196] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/8b55b2f9005e4a72b8499a6e214f8017
I20260928 15:55:45.618953 10734 ts_tablet_manager.cc:1939] T 8b55b2f9005e4a72b8499a6e214f8017 P 22beb3125f97445e945991427a503db2: Deleting consensus metadata
I20260928 15:55:45.619954 10733 ts_tablet_manager.cc:1918] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:45.620208 10733 log.cc:1196] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790610618730370-23159-0/minicluster-data/ts-0/wal/wals/8b55b2f9005e4a72b8499a6e214f8017
I20260928 15:55:45.620862 10733 ts_tablet_manager.cc:1939] T 8b55b2f9005e4a72b8499a6e214f8017 P aaad221777f44dc1a92bc116b7ff4ce0: Deleting consensus metadata
I20260928 15:55:45.620937 10216 catalog_manager.cc:4841] TS 22beb3125f97445e945991427a503db2 (127.22.157.195:39243): tablet 8b55b2f9005e4a72b8499a6e214f8017 (table test-table [id=add310a217fc43429bfdcf9b6b3836ad]) successfully deleted
I20260928 15:55:45.622263 10216 catalog_manager.cc:4841] TS aaad221777f44dc1a92bc116b7ff4ce0 (127.22.157.193:33015): tablet 8b55b2f9005e4a72b8499a6e214f8017 (table test-table [id=add310a217fc43429bfdcf9b6b3836ad]) successfully deleted
Sep 28 15:55:45 dist-test-slave-1vw0 krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.193@KRBTEST.COM
I20260928 15:55:45.634231 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.592385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.22.157.193:33015)
0928 15:55:45.592626 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.592636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.606143 (+ 13507us) server_negotiation.cc:407] Connection header received
0928 15:55:45.606354 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.606358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.606447 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.606563 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:45.618376 (+ 11813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.619732 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.622551 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.623033 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.623148 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.627998 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:45.628011 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:45.628029 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:45.628068 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:45.631206 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.631968 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.631979 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.631981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.632149 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.632541 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.632545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.632547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.632965 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:45.633318 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.633601 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.633860 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 15:55:45.634958 10416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:45.749980 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 10293
I20260928 15:55:45.769940 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 10427
Sep 28 15:55:45 dist-test-slave-1vw0 krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.194@KRBTEST.COM
I20260928 15:55:45.788918 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.770901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.22.157.194:44075)
0928 15:55:45.771097 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.771101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.771172 (+ 71us) server_negotiation.cc:407] Connection header received
0928 15:55:45.771350 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.771354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.771425 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.771510 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:45.772901 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.773652 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.774953 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.775324 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.775417 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.780309 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:45.780330 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:45.780371 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:45.780425 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:45.785340 (+ 4915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.786393 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.786406 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.786411 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.786600 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.787025 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.787031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.787035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.787582 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:45.788028 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.788258 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.788500 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:55:45.789639 10550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:45.881500 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 10427
I20260928 15:55:45.899433 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 10561
Sep 28 15:55:45 dist-test-slave-1vw0 krb5kdc[10187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.195@KRBTEST.COM
I20260928 15:55:45.915313 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:45.900212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.22.157.195:39243)
0928 15:55:45.900452 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:45.900458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:45.900479 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:55:45.900666 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:45.900671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:45.900797 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:45.900927 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:45.902307 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.903090 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.904376 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:45.904923 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:45.905061 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:45.909852 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:45.909869 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:45.909895 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:45.909932 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:45.912591 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.913354 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.913362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.913364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.913546 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:45.913919 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:45.913922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:45.913925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:45.914341 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:45.914584 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:45.914847 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 15:55:45.915045 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 15:55:45.916074 10684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:46.009738 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 10561
I20260928 15:55:46.027413 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 10200
I20260928 15:55:46.028638 10262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:46.140933 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 10200
2026-09-28T15:55:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4560 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T15:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:46Z Disabled control of system clock
I20260928 15:55:46.227941 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:33405
--webserver_interface=127.22.157.254
--webserver_port=0
--builtin_ntp_servers=127.22.157.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:55:46.568110 10743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:46.568488 10743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:46.568645 10743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:46.576105 10743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:46.576282 10743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:46.576375 10743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:46.576488 10743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:46.576579 10743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:46.586516 10743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:33405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:46.587898 10743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:46.589847 10743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:46.601783 10748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:46.601908 10749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:46.601931 10751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:46.603448 10743 server_base.cc:1034] running on GCE node
I20260928 15:55:46.604331 10743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:46.605979 10743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:46.607223 10743 hybrid_clock.cc:648] HybridClock initialized: now 1790610946607183 us; error 48 us; skew 500 ppm
I20260928 15:55:46.607604 10743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:46.610033 10743 webserver.cc:460] Webserver started at http://127.22.157.254:44795/ using document root <none> and password file <none>
I20260928 15:55:46.610765 10743 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:46.610900 10743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:46.611246 10743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:46.614428 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "626fad28290b4729be2819020012d0ae"
format_stamp: "Formatted at 2026-09-28 15:55:46 on dist-test-slave-1vw0"
server_key: "a814d47ed8933ddbc62203ddf94411b9"
server_key_iv: "3d4c728193c5e306e1a238dbba4c7b30"
server_key_version: "encryptionkey@0"
I20260928 15:55:46.615320 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "626fad28290b4729be2819020012d0ae"
format_stamp: "Formatted at 2026-09-28 15:55:46 on dist-test-slave-1vw0"
server_key: "a814d47ed8933ddbc62203ddf94411b9"
server_key_iv: "3d4c728193c5e306e1a238dbba4c7b30"
server_key_version: "encryptionkey@0"
I20260928 15:55:46.620908 10743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:55:46.624872 10757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:46.626442 10743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:55:46.626664 10743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "626fad28290b4729be2819020012d0ae"
format_stamp: "Formatted at 2026-09-28 15:55:46 on dist-test-slave-1vw0"
server_key: "a814d47ed8933ddbc62203ddf94411b9"
server_key_iv: "3d4c728193c5e306e1a238dbba4c7b30"
server_key_version: "encryptionkey@0"
I20260928 15:55:46.626925 10743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:46.645226 10743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:46.646250 10743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:46.646589 10743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:46.667014 10743 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:33405
I20260928 15:55:46.667026 10808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:33405 every 8 connection(s)
I20260928 15:55:46.668510 10743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:46.672533 10809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:46.676390 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 10743
I20260928 15:55:46.676551 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:46.677069 23159 external_mini_cluster.cc:1428] Setting key 823efe54f2b917f1ec0829f7d36e3b93
I20260928 15:55:46.684499 10809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae: Bootstrap starting.
I20260928 15:55:46.688587 10809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:46.690024 10809 log.cc:824] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:46.693118 10809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae: No bootstrap required, opened a new log
I20260928 15:55:46.697513 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:46.679256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.22.157.254:33405)
0928 15:55:46.680347 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:46.680359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:46.680402 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:55:46.682913 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:46.682940 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:46.683510 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:46.683943 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:46.684787 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:46.686352 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:46.688350 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:46.689289 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:46.689444 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:46.694786 (+ 5342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:46.694794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:46.694813 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:46.695427 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:46.695516 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:46.695584 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:46.695681 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:46.695960 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 15:55:46.696348 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":392,"threads_started":1}
I20260928 15:55:46.701689 10809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626fad28290b4729be2819020012d0ae" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33405 } }
I20260928 15:55:46.702054 10809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:46.702208 10809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626fad28290b4729be2819020012d0ae, State: Initialized, Role: FOLLOWER
I20260928 15:55:46.702792 10809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626fad28290b4729be2819020012d0ae" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33405 } }
I20260928 15:55:46.703043 10809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:46.703215 10809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:46.703398 10809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:46.706187 10809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626fad28290b4729be2819020012d0ae" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33405 } }
I20260928 15:55:46.706830 10809 leader_election.cc:302] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 626fad28290b4729be2819020012d0ae; no voters:
I20260928 15:55:46.707477 10809 leader_election.cc:288] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:46.707665 10814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:46.708568 10814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [term 1 LEADER]: Becoming Leader. State: Replica: 626fad28290b4729be2819020012d0ae, State: Running, Role: LEADER
I20260928 15:55:46.709192 10814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626fad28290b4729be2819020012d0ae" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33405 } }
I20260928 15:55:46.709765 10809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:46.712245 10816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 626fad28290b4729be2819020012d0ae. Latest consensus state: current_term: 1 leader_uuid: "626fad28290b4729be2819020012d0ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626fad28290b4729be2819020012d0ae" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33405 } } }
I20260928 15:55:46.712637 10816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:46.711966 10815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "626fad28290b4729be2819020012d0ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626fad28290b4729be2819020012d0ae" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 33405 } } }
I20260928 15:55:46.713272 10815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:46.716670 10820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:46.722276 10820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:46.731706 10820 catalog_manager.cc:1347] Generated new cluster ID: 9ed4c77c95444d69b9009de7bc0db65d
I20260928 15:55:46.731809 10820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:46.742465 10820 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:46.743854 10820 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:46.752282 10820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae: Generated new TSK 0
I20260928 15:55:46.753309 10820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:55:46.761708 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.193:0
--local_ip_for_outbound_sockets=127.22.157.193
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:33405
--builtin_ntp_servers=127.22.157.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 15:55:47.125190 10834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:47.125496 10834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:47.125584 10834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:47.133502 10834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:47.133625 10834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:47.133746 10834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.193
I20260928 15:55:47.143361 10834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.157.193
--webserver_port=0
--tserver_master_addrs=127.22.157.254:33405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.193
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:47.144801 10834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:47.146703 10834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:47.160394 10839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:47.160382 10840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:47.160635 10842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:47.161609 10834 server_base.cc:1034] running on GCE node
I20260928 15:55:47.162377 10834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:47.163544 10834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:47.164842 10834 hybrid_clock.cc:648] HybridClock initialized: now 1790610947164771 us; error 59 us; skew 500 ppm
I20260928 15:55:47.165202 10834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:47.167572 10834 webserver.cc:460] Webserver started at http://127.22.157.193:36093/ using document root <none> and password file <none>
I20260928 15:55:47.168309 10834 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:47.168443 10834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:47.168816 10834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:47.172014 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/data/instance:
uuid: "abb0a5f69762425eb37722733fdfc091"
format_stamp: "Formatted at 2026-09-28 15:55:47 on dist-test-slave-1vw0"
server_key: "0d1dacb86f53d258633cec28e4fe2497"
server_key_iv: "775845f98786f4b5bab3853e6702098e"
server_key_version: "encryptionkey@0"
I20260928 15:55:47.172932 10834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance:
uuid: "abb0a5f69762425eb37722733fdfc091"
format_stamp: "Formatted at 2026-09-28 15:55:47 on dist-test-slave-1vw0"
server_key: "0d1dacb86f53d258633cec28e4fe2497"
server_key_iv: "775845f98786f4b5bab3853e6702098e"
server_key_version: "encryptionkey@0"
I20260928 15:55:47.178310 10834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 15:55:47.182044 10848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:47.183509 10834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:55:47.183688 10834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/wal
uuid: "abb0a5f69762425eb37722733fdfc091"
format_stamp: "Formatted at 2026-09-28 15:55:47 on dist-test-slave-1vw0"
server_key: "0d1dacb86f53d258633cec28e4fe2497"
server_key_iv: "775845f98786f4b5bab3853e6702098e"
server_key_version: "encryptionkey@0"
I20260928 15:55:47.183912 10834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:47.215857 10834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:47.216980 10834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:47.217303 10834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:47.218293 10834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:47.219849 10834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:47.219980 10834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:47.220110 10834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:47.220204 10834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:47.255594 10834 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.193:42023
I20260928 15:55:47.255686 10960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.193:42023 every 8 connection(s)
I20260928 15:55:47.257020 10834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/data/info.pb
I20260928 15:55:47.263136 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 10834
I20260928 15:55:47.263334 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-0/wal/instance
I20260928 15:55:47.263968 23159 external_mini_cluster.cc:1428] Setting key 273786924579f8724916c602ced40ebd
I20260928 15:55:47.267123 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.194:0
--local_ip_for_outbound_sockets=127.22.157.194
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:33405
--builtin_ntp_servers=127.22.157.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:55:47.271867 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:47.260057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:51697 (local address 127.22.157.254:33405)
0928 15:55:47.260554 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:47.260561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:47.261916 (+ 1355us) server_negotiation.cc:407] Connection header received
0928 15:55:47.264634 (+ 2718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:47.264640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:47.264780 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:47.264916 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:47.266474 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:47.267256 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:47.268988 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:47.269414 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:47.269555 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:47.271000 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:47.271005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:47.271008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:47.271029 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:47.271069 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:47.271082 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:47.271188 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:47.271411 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 15:55:47.271517 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":229,"threads_started":1}
I20260928 15:55:47.273061 10964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:47.260625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:33405 (local address 127.22.157.193:51697)
0928 15:55:47.261743 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:47.261784 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:47.264239 (+ 2455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:47.265285 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:47.265295 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:47.265404 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:47.266283 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:47.266314 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:47.267458 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:47.267462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:47.268846 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:47.268858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:47.269679 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:47.269689 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:47.269930 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:47.270666 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:55:47.270668 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:47.270711 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:55:47.270795 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:47.271189 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:47.271202 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:47.271213 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:55:47.271402 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 15:55:47.271834 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":306,"threads_started":1}
I20260928 15:55:47.274660 10961 heartbeater.cc:347] Connected to a master server at 127.22.157.254:33405
I20260928 15:55:47.275029 10961 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:47.276212 10961 heartbeater.cc:511] Master 127.22.157.254:33405 requested a full tablet report, sending...
I20260928 15:55:47.278913 10774 ts_manager.cc:194] Registered new tserver with Master: abb0a5f69762425eb37722733fdfc091 (127.22.157.193:42023)
I20260928 15:55:47.281167 10774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.193:51697
W20260928 15:55:47.612566 10965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:47.612946 10965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:47.613039 10965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:47.620604 10965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:47.620744 10965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:47.620855 10965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.194
I20260928 15:55:47.634394 10965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.157.194
--webserver_port=0
--tserver_master_addrs=127.22.157.254:33405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.194
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:47.636264 10965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:47.638764 10965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:47.656991 10970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:47.657011 10971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:47.657475 10973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:47.658048 10965 server_base.cc:1034] running on GCE node
I20260928 15:55:47.658928 10965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:47.660254 10965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:47.661542 10965 hybrid_clock.cc:648] HybridClock initialized: now 1790610947661477 us; error 50 us; skew 500 ppm
I20260928 15:55:47.661948 10965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:47.665211 10965 webserver.cc:460] Webserver started at http://127.22.157.194:40089/ using document root <none> and password file <none>
I20260928 15:55:47.666266 10965 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:47.666393 10965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:47.666764 10965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:47.670696 10965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/data/instance:
uuid: "e28189d4fd3742469da4651a10196f57"
format_stamp: "Formatted at 2026-09-28 15:55:47 on dist-test-slave-1vw0"
server_key: "e6034ed49bc052aab20140195e250471"
server_key_iv: "0ff35da6c167e46c54e88cabe8609daf"
server_key_version: "encryptionkey@0"
I20260928 15:55:47.671710 10965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance:
uuid: "e28189d4fd3742469da4651a10196f57"
format_stamp: "Formatted at 2026-09-28 15:55:47 on dist-test-slave-1vw0"
server_key: "e6034ed49bc052aab20140195e250471"
server_key_iv: "0ff35da6c167e46c54e88cabe8609daf"
server_key_version: "encryptionkey@0"
I20260928 15:55:47.679173 10965 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260928 15:55:47.684057 10979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:47.686005 10965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 15:55:47.686239 10965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/wal
uuid: "e28189d4fd3742469da4651a10196f57"
format_stamp: "Formatted at 2026-09-28 15:55:47 on dist-test-slave-1vw0"
server_key: "e6034ed49bc052aab20140195e250471"
server_key_iv: "0ff35da6c167e46c54e88cabe8609daf"
server_key_version: "encryptionkey@0"
I20260928 15:55:47.686506 10965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:47.700529 10965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:47.701398 10965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:47.701656 10965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:47.702593 10965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:47.704025 10965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:47.704108 10965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:47.704190 10965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:47.704245 10965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:47.739596 10965 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.194:39431
I20260928 15:55:47.739701 11091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.194:39431 every 8 connection(s)
I20260928 15:55:47.740998 10965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/data/info.pb
I20260928 15:55:47.749993 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 10965
I20260928 15:55:47.750214 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/wal/instance
I20260928 15:55:47.750816 23159 external_mini_cluster.cc:1428] Setting key cc2964feb1ea7880982b6a33740f2e5b
I20260928 15:55:47.754487 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.157.195:0
--local_ip_for_outbound_sockets=127.22.157.195
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:33405
--builtin_ntp_servers=127.22.157.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 15:55:47.755913 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:47.743756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:55177 (local address 127.22.157.254:33405)
0928 15:55:47.743985 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:47.743991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:47.745440 (+ 1449us) server_negotiation.cc:407] Connection header received
0928 15:55:47.747355 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:47.747360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:47.747433 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:47.747524 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:47.749136 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:47.749997 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:47.752339 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:47.753154 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:47.753287 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:47.755066 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:47.755071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:47.755074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:47.755095 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:47.755130 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:47.755143 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:47.755227 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:47.755546 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 15:55:47.755660 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 15:55:47.757639 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:47.744257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:33405 (local address 127.22.157.194:55177)
0928 15:55:47.745275 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:47.745322 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:47.747092 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:47.747843 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:47.747852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:47.747953 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:47.748910 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:47.748935 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:47.750212 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:47.750218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:47.752176 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:47.752190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:47.753409 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:47.753417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:47.753737 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:47.754612 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:55:47.754617 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:47.754688 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:55:47.754829 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:47.755254 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:47.755269 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:47.755282 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 15:55:47.755509 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 15:55:47.756034 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":290,"threads_started":1}
I20260928 15:55:47.759446 11092 heartbeater.cc:347] Connected to a master server at 127.22.157.254:33405
I20260928 15:55:47.759775 11092 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:47.760557 11092 heartbeater.cc:511] Master 127.22.157.254:33405 requested a full tablet report, sending...
I20260928 15:55:47.762405 10774 ts_manager.cc:194] Registered new tserver with Master: e28189d4fd3742469da4651a10196f57 (127.22.157.194:39431)
I20260928 15:55:47.763374 10774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.194:55177
W20260928 15:55:48.111897 11095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:48.112191 11095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:48.112278 11095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:48.119771 11095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:48.119880 11095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:48.119990 11095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.157.195
I20260928 15:55:48.135931 11095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.157.195
--webserver_port=0
--tserver_master_addrs=127.22.157.254:33405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.157.195
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:48.137831 11095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:48.140611 11095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:48.158528 11101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:48.159421 11103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:48.162006 11100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:48.166498 11095 server_base.cc:1034] running on GCE node
I20260928 15:55:48.167557 11095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:48.169113 11095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:48.170423 11095 hybrid_clock.cc:648] HybridClock initialized: now 1790610948170339 us; error 76 us; skew 500 ppm
I20260928 15:55:48.170960 11095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 15:55:48.174470 11095 webserver.cc:460] Webserver started at http://127.22.157.195:43795/ using document root <none> and password file <none>
I20260928 15:55:48.175614 11095 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:48.175768 11095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:48.176204 11095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:48.181022 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/data/instance:
uuid: "3bca4275847a40f698698d5721002566"
format_stamp: "Formatted at 2026-09-28 15:55:48 on dist-test-slave-1vw0"
server_key: "0471b1c80b99af6341261bc9568e883f"
server_key_iv: "5f51835ec86ab95387a9b3289706108f"
server_key_version: "encryptionkey@0"
I20260928 15:55:48.182224 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance:
uuid: "3bca4275847a40f698698d5721002566"
format_stamp: "Formatted at 2026-09-28 15:55:48 on dist-test-slave-1vw0"
server_key: "0471b1c80b99af6341261bc9568e883f"
server_key_iv: "5f51835ec86ab95387a9b3289706108f"
server_key_version: "encryptionkey@0"
I20260928 15:55:48.190405 11095 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260928 15:55:48.196058 11109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:48.198031 11095 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 15:55:48.198232 11095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/wal
uuid: "3bca4275847a40f698698d5721002566"
format_stamp: "Formatted at 2026-09-28 15:55:48 on dist-test-slave-1vw0"
server_key: "0471b1c80b99af6341261bc9568e883f"
server_key_iv: "5f51835ec86ab95387a9b3289706108f"
server_key_version: "encryptionkey@0"
I20260928 15:55:48.198551 11095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:48.238698 11095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:48.239943 11095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:48.240341 11095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:48.241832 11095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 15:55:48.244139 11095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 15:55:48.244277 11095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:48.244401 11095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 15:55:48.244504 11095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:48.284780 10961 heartbeater.cc:503] Master 127.22.157.254:33405 was elected leader, sending a full tablet report...
I20260928 15:55:48.298058 11095 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.195:43833
I20260928 15:55:48.298113 11221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.195:43833 every 8 connection(s)
I20260928 15:55:48.299405 11095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/data/info.pb
I20260928 15:55:48.299779 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 11095
I20260928 15:55:48.299979 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/wal/instance
I20260928 15:55:48.300643 23159 external_mini_cluster.cc:1428] Setting key 2e5b9be221b385496b0c31e37ca4a215
I20260928 15:55:48.319396 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.302754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:35941 (local address 127.22.157.254:33405)
0928 15:55:48.305760 (+ 3006us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.305770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.306577 (+ 807us) server_negotiation.cc:407] Connection header received
0928 15:55:48.309923 (+ 3346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.309928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.310011 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.310156 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:48.311809 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.312625 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.315692 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.316331 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.316471 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.318460 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:48.318465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:48.318468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:48.318487 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:48.318528 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:48.318587 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:48.318663 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.319003 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.319095 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2818,"thread_start_us":2695,"threads_started":1}
I20260928 15:55:48.321169 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.303653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:33405 (local address 127.22.157.195:35941)
0928 15:55:48.306379 (+ 2726us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.306445 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.309504 (+ 3059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.310463 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.310480 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.310649 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:55:48.311583 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.311612 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.312853 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.312861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.315526 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.315546 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.316816 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.316829 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.317218 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.318042 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 15:55:48.318048 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:55:48.318111 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 15:55:48.318243 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:55:48.318694 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:55:48.318706 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:55:48.318717 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.318922 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.319447 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"thread_start_us":1248,"threads_started":1}
I20260928 15:55:48.322816 11222 heartbeater.cc:347] Connected to a master server at 127.22.157.254:33405
I20260928 15:55:48.323169 11222 heartbeater.cc:464] Registering TS with master...
I20260928 15:55:48.324028 11222 heartbeater.cc:511] Master 127.22.157.254:33405 requested a full tablet report, sending...
I20260928 15:55:48.325691 10774 ts_manager.cc:194] Registered new tserver with Master: 3bca4275847a40f698698d5721002566 (127.22.157.195:43833)
I20260928 15:55:48.326536 10774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.157.195:35941
I20260928 15:55:48.330477 23159 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 15:55:48.347338 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.336813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.22.157.254:33405)
0928 15:55:48.337081 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.337091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.341398 (+ 4307us) server_negotiation.cc:407] Connection header received
0928 15:55:48.341639 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.341644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.341722 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.341840 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:48.342409 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.343143 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.344536 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.345044 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.345190 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.346492 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:48.346500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:48.346508 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:48.346540 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:48.346586 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:48.346605 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:48.346724 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.346877 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.346993 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 15:55:48.352238 10774 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 15:55:48.359916 10774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 15:55:48.402243 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.375700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.22.157.195:43833)
0928 15:55:48.382880 (+ 7180us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.382894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.389663 (+ 6769us) server_negotiation.cc:407] Connection header received
0928 15:55:48.390340 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.390356 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.390767 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.391007 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:48.392440 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.394474 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.397957 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.399546 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.399717 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.399978 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.401604 (+ 1626us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.401816 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6978,"thread_start_us":6824,"threads_started":1}
I20260928 15:55:48.403505 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.375291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:43833 (local address 127.0.0.1:58090)
0928 15:55:48.389461 (+ 14170us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.389497 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.390043 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.391533 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.391546 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.391593 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:48.392156 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.392173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.394668 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.394673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.397704 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.397719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.400161 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.400170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.401291 (+ 1121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.401370 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.402793 (+ 1423us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.403046 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13986,"mutex_wait_us":3,"thread_start_us":14409,"threads_started":1}
I20260928 15:55:48.405305 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.374085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.22.157.193:42023)
0928 15:55:48.392340 (+ 18255us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.392361 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.392401 (+ 40us) server_negotiation.cc:407] Connection header received
0928 15:55:48.392517 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.392528 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.393191 (+ 663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.394270 (+ 1079us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:48.394580 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.398151 (+ 3571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.401159 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.402836 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.404377 (+ 1541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.404629 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.404702 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.404917 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18009,"thread_start_us":18635,"threads_started":1}
I20260928 15:55:48.403986 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.373393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.0.0.1:60262)
0928 15:55:48.374775 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.374837 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.375059 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.393464 (+ 18405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.393471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.393508 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:48.393898 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.393912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.398327 (+ 4415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.398335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.400961 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.400974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.403083 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.403089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.403296 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.403346 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.403417 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.403583 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":131,"thread_start_us":939,"threads_started":1}
I20260928 15:55:48.406307 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.373931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:39431 (local address 127.0.0.1:52642)
0928 15:55:48.388964 (+ 15033us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.389001 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.389196 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.395118 (+ 5922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.395122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.395148 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:48.395576 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.395589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.397724 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.397729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.399861 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.399876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.403061 (+ 3185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.403071 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.403387 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.403481 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.403603 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.405402 (+ 1799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14832,"mutex_wait_us":25,"thread_start_us":14600,"threads_started":1}
I20260928 15:55:48.407903 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.374542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.22.157.194:39431)
0928 15:55:48.394220 (+ 19678us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.394229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.394263 (+ 34us) server_negotiation.cc:407] Connection header received
0928 15:55:48.394358 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.394366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.394664 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.395336 (+ 672us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:48.395773 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.397537 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.400920 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.402749 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.405574 (+ 2825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.406590 (+ 1016us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.406682 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.406903 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19467,"thread_start_us":20281,"threads_started":1}
I20260928 15:55:48.408582 11157 tablet_service.cc:1467] Processing CreateTablet for tablet 32f26e387271420fa857646aef3d9773 (DEFAULT_TABLE table=test-table [id=634e62dc50f7435292fbcd5e0742d627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:48.410611 11157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f26e387271420fa857646aef3d9773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:48.410392 10896 tablet_service.cc:1467] Processing CreateTablet for tablet 32f26e387271420fa857646aef3d9773 (DEFAULT_TABLE table=test-table [id=634e62dc50f7435292fbcd5e0742d627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:48.412467 10896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f26e387271420fa857646aef3d9773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:48.412324 11027 tablet_service.cc:1467] Processing CreateTablet for tablet 32f26e387271420fa857646aef3d9773 (DEFAULT_TABLE table=test-table [id=634e62dc50f7435292fbcd5e0742d627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 15:55:48.413887 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f26e387271420fa857646aef3d9773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:48.428511 11241 tablet_bootstrap.cc:492] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: Bootstrap starting.
I20260928 15:55:48.432510 11241 tablet_bootstrap.cc:654] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:48.434111 11241 log.cc:824] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:48.436851 11243 tablet_bootstrap.cc:492] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091: Bootstrap starting.
I20260928 15:55:48.441542 11242 tablet_bootstrap.cc:492] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: Bootstrap starting.
I20260928 15:55:48.443292 11243 tablet_bootstrap.cc:654] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:48.444629 11243 log.cc:824] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:48.445405 11242 tablet_bootstrap.cc:654] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:48.445515 11241 tablet_bootstrap.cc:492] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: No bootstrap required, opened a new log
I20260928 15:55:48.445801 11241 ts_tablet_manager.cc:1397] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20260928 15:55:48.446686 11242 log.cc:824] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:48.448120 11243 tablet_bootstrap.cc:492] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091: No bootstrap required, opened a new log
I20260928 15:55:48.448400 11243 ts_tablet_manager.cc:1397] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260928 15:55:48.449450 11242 tablet_bootstrap.cc:492] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: No bootstrap required, opened a new log
I20260928 15:55:48.449733 11242 ts_tablet_manager.cc:1397] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 15:55:48.457010 11241 raft_consensus.cc:348] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.457571 11241 raft_consensus.cc:374] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:48.457674 11241 raft_consensus.cc:729] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bca4275847a40f698698d5721002566, State: Initialized, Role: FOLLOWER
I20260928 15:55:48.457871 11242 raft_consensus.cc:348] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.458348 11242 raft_consensus.cc:374] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:48.458457 11242 raft_consensus.cc:729] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28189d4fd3742469da4651a10196f57, State: Initialized, Role: FOLLOWER
I20260928 15:55:48.458376 11241 consensus_queue.cc:260] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.459187 11242 consensus_queue.cc:260] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.459365 11243 raft_consensus.cc:348] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.459930 11243 raft_consensus.cc:374] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:48.460067 11243 raft_consensus.cc:729] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb0a5f69762425eb37722733fdfc091, State: Initialized, Role: FOLLOWER
I20260928 15:55:48.460640 11092 heartbeater.cc:503] Master 127.22.157.254:33405 was elected leader, sending a full tablet report...
I20260928 15:55:48.461177 11242 ts_tablet_manager.cc:1428] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260928 15:55:48.461938 11243 consensus_queue.cc:260] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.463941 11243 ts_tablet_manager.cc:1428] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260928 15:55:48.465010 11222 heartbeater.cc:503] Master 127.22.157.254:33405 was elected leader, sending a full tablet report...
I20260928 15:55:48.465822 11241 ts_tablet_manager.cc:1428] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
W20260928 15:55:48.493176 11093 tablet.cc:2369] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:55:48.510012 10962 tablet.cc:2369] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 15:55:48.552891 11223 tablet.cc:2369] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:55:48.914850 11248 raft_consensus.cc:482] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:55:48.915132 11248 raft_consensus.cc:504] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.916715 11248 leader_election.cc:288] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abb0a5f69762425eb37722733fdfc091 (127.22.157.193:42023), 3bca4275847a40f698698d5721002566 (127.22.157.195:43833)
I20260928 15:55:48.918840 11247 raft_consensus.cc:482] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:55:48.919153 11247 raft_consensus.cc:504] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.925189 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.917241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.22.157.194:58693)
0928 15:55:48.917836 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.917853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.918018 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.918652 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.918657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.918692 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:48.919096 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.919113 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.920359 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.920365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.922348 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.922365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.924249 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.924255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.924448 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.924484 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.924577 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.924752 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"spinlock_wait_cycles":33280,"thread_start_us":333,"threads_started":1}
I20260928 15:55:48.925818 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.918079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:43833 (local address 127.22.157.194:45787)
0928 15:55:48.918681 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.918697 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.918819 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.919176 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.919183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.919215 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:48.919575 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.919589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.921061 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.921066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.923123 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.923138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.924981 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.924986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.925186 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.925229 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.925321 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.925452 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":1,"spinlock_wait_cycles":1408,"thread_start_us":274,"threads_started":1}
I20260928 15:55:48.925963 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.917233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:58693 (local address 127.22.157.193:42023)
0928 15:55:48.917879 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.917887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.917917 (+ 30us) server_negotiation.cc:407] Connection header received
0928 15:55:48.918194 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.918198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.918316 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.918428 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:48.919306 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.920216 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.922592 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.923891 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.924070 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.924157 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.925449 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.925599 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":339,"threads_started":1}
I20260928 15:55:48.926468 11247 leader_election.cc:288] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abb0a5f69762425eb37722733fdfc091 (127.22.157.193:42023), e28189d4fd3742469da4651a10196f57 (127.22.157.194:39431)
I20260928 15:55:48.926648 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.918208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.194:45787 (local address 127.22.157.195:43833)
0928 15:55:48.918842 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.918848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.918871 (+ 23us) server_negotiation.cc:407] Connection header received
0928 15:55:48.918943 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.918949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.919027 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.919163 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:48.919749 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.920916 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.923354 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.924615 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.924794 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.924857 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.926026 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.926169 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":430,"threads_started":1}
I20260928 15:55:48.927721 11177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f26e387271420fa857646aef3d9773" candidate_uuid: "e28189d4fd3742469da4651a10196f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bca4275847a40f698698d5721002566" is_pre_election: true
I20260928 15:55:48.928396 11177 raft_consensus.cc:2454] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e28189d4fd3742469da4651a10196f57 in term 0.
I20260928 15:55:48.929355 10982 leader_election.cc:302] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bca4275847a40f698698d5721002566, e28189d4fd3742469da4651a10196f57; no voters:
I20260928 15:55:48.929979 11248 raft_consensus.cc:2781] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 15:55:48.930141 11248 raft_consensus.cc:482] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 15:55:48.930238 11248 raft_consensus.cc:3037] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:48.933684 11248 raft_consensus.cc:504] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.934489 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.921954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:48077 (local address 127.22.157.193:42023)
0928 15:55:48.926193 (+ 4239us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.926198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.927218 (+ 1020us) server_negotiation.cc:407] Connection header received
0928 15:55:48.927523 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.927533 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.927592 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.927641 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:48.928288 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.929320 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.931935 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.933236 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.933384 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.933477 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.934057 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.934215 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4127,"threads_started":1}
I20260928 15:55:48.935216 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.921502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.22.157.195:48077)
0928 15:55:48.927100 (+ 5598us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.927125 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.927362 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.927773 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.927776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.927806 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:48.928077 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.928083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.929493 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.929501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.931642 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.931666 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.933687 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.933693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.933860 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.933896 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.934686 (+ 790us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.934830 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5441,"thread_start_us":5464,"threads_started":1}
I20260928 15:55:48.935452 11248 leader_election.cc:288] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [CANDIDATE]: Term 1 election: Requested vote from peers abb0a5f69762425eb37722733fdfc091 (127.22.157.193:42023), 3bca4275847a40f698698d5721002566 (127.22.157.195:43833)
I20260928 15:55:48.935520 11177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f26e387271420fa857646aef3d9773" candidate_uuid: "e28189d4fd3742469da4651a10196f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bca4275847a40f698698d5721002566"
I20260928 15:55:48.935786 11177 raft_consensus.cc:3037] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:48.936676 11249 raft_consensus.cc:482] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 15:55:48.937022 11249 raft_consensus.cc:504] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.937448 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f26e387271420fa857646aef3d9773" candidate_uuid: "3bca4275847a40f698698d5721002566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb0a5f69762425eb37722733fdfc091" is_pre_election: true
I20260928 15:55:48.937927 10916 raft_consensus.cc:2454] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bca4275847a40f698698d5721002566 in term 0.
I20260928 15:55:48.938657 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f26e387271420fa857646aef3d9773" candidate_uuid: "e28189d4fd3742469da4651a10196f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb0a5f69762425eb37722733fdfc091" is_pre_election: true
I20260928 15:55:48.938867 10916 raft_consensus.cc:2454] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e28189d4fd3742469da4651a10196f57 in term 0.
I20260928 15:55:48.939064 11249 leader_election.cc:288] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bca4275847a40f698698d5721002566 (127.22.157.195:43833), e28189d4fd3742469da4651a10196f57 (127.22.157.194:39431)
I20260928 15:55:48.939663 10916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f26e387271420fa857646aef3d9773" candidate_uuid: "e28189d4fd3742469da4651a10196f57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb0a5f69762425eb37722733fdfc091"
I20260928 15:55:48.939756 11112 leader_election.cc:302] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bca4275847a40f698698d5721002566, abb0a5f69762425eb37722733fdfc091; no voters:
I20260928 15:55:48.939854 10916 raft_consensus.cc:3037] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:48.941860 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.921556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:39431 (local address 127.22.157.195:48383)
0928 15:55:48.927313 (+ 5757us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.927329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.927418 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.935017 (+ 7599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.935026 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.935073 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:48.935553 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.935568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.937021 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.937027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.939207 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.939224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.941064 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.941070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.941264 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.941302 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.941377 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.941518 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5632,"spinlock_wait_cycles":1792,"thread_start_us":5607,"threads_started":1}
I20260928 15:55:48.942498 11177 raft_consensus.cc:2454] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e28189d4fd3742469da4651a10196f57 in term 1.
I20260928 15:55:48.942639 11247 raft_consensus.cc:2741] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 15:55:48.942816 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.921945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.195:48383 (local address 127.22.157.194:39431)
0928 15:55:48.934456 (+ 12511us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.934466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.934512 (+ 46us) server_negotiation.cc:407] Connection header received
0928 15:55:48.934595 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.934603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.934704 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.935545 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:48.935865 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.936843 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.939475 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.940853 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.942164 (+ 1311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.942255 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.942307 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.942437 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12292,"thread_start_us":12885,"threads_started":1}
I20260928 15:55:48.943466 10916 raft_consensus.cc:2454] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e28189d4fd3742469da4651a10196f57 in term 1.
I20260928 15:55:48.943673 11047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f26e387271420fa857646aef3d9773" candidate_uuid: "3bca4275847a40f698698d5721002566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28189d4fd3742469da4651a10196f57" is_pre_election: true
I20260928 15:55:48.944311 11047 raft_consensus.cc:2379] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3bca4275847a40f698698d5721002566 in current term 1: Already voted for candidate e28189d4fd3742469da4651a10196f57 in this term.
I20260928 15:55:48.944620 10982 leader_election.cc:302] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bca4275847a40f698698d5721002566, e28189d4fd3742469da4651a10196f57; no voters:
I20260928 15:55:48.945158 11248 raft_consensus.cc:2781] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:48.952415 11248 raft_consensus.cc:686] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 1 LEADER]: Becoming Leader. State: Replica: e28189d4fd3742469da4651a10196f57, State: Running, Role: LEADER
I20260928 15:55:48.953141 11248 consensus_queue.cc:237] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } }
I20260928 15:55:48.959801 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.939843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:35503 (local address 127.22.157.195:43833)
0928 15:55:48.943394 (+ 3551us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.943402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.951478 (+ 8076us) server_negotiation.cc:407] Connection header received
0928 15:55:48.952101 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.952109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.952225 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.952304 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:48.953063 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.954126 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.957435 (+ 3309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.958644 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.958774 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.958842 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.959368 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.959505 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3430}
I20260928 15:55:48.959982 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.939408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.195:43833 (local address 127.22.157.193:35503)
0928 15:55:48.951332 (+ 11924us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.951367 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.951897 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.952506 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.952511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.952542 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:48.952884 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.952897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.954343 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.954350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.957058 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.957073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.959053 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.959057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.959200 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.959223 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.959324 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.959601 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11738,"mutex_wait_us":115,"thread_start_us":12092,"threads_started":1}
I20260928 15:55:48.959491 10774 catalog_manager.cc:5495] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 reported cstate change: term changed from 0 to 1, leader changed from <none> to e28189d4fd3742469da4651a10196f57 (127.22.157.194). New cstate: current_term: 1 leader_uuid: "e28189d4fd3742469da4651a10196f57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e28189d4fd3742469da4651a10196f57" member_type: VOTER last_known_addr { host: "127.22.157.194" port: 39431 } health_report { overall_health: HEALTHY } } }
I20260928 15:55:48.960780 11177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f26e387271420fa857646aef3d9773" candidate_uuid: "abb0a5f69762425eb37722733fdfc091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bca4275847a40f698698d5721002566" is_pre_election: true
I20260928 15:55:48.961086 11177 raft_consensus.cc:2379] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate abb0a5f69762425eb37722733fdfc091 in current term 1: Already voted for candidate e28189d4fd3742469da4651a10196f57 in this term.
I20260928 15:55:48.963815 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.942597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.157.193:37385 (local address 127.22.157.194:39431)
0928 15:55:48.943040 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:48.943048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:48.955889 (+ 12841us) server_negotiation.cc:407] Connection header received
0928 15:55:48.956766 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:48.956776 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:48.956870 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:48.956985 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 15:55:48.957755 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.958825 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.962005 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.962981 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.963142 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.963214 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:48.963428 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 15:55:48.963554 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":14269,"threads_started":1}
I20260928 15:55:48.963867 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:48.945997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.194:39431 (local address 127.22.157.193:37385)
0928 15:55:48.955800 (+ 9803us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:48.955818 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:55:48.956536 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:55:48.957132 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:55:48.957136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:55:48.957164 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 15:55:48.957560 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.957573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.958986 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.958990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.961476 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:55:48.961496 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:55:48.963143 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:55:48.963147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:55:48.963262 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:48.963285 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 15:55:48.963390 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 15:55:48.963515 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9635,"thread_start_us":9503,"threads_started":1}
I20260928 15:55:48.964795 11047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f26e387271420fa857646aef3d9773" candidate_uuid: "abb0a5f69762425eb37722733fdfc091" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28189d4fd3742469da4651a10196f57" is_pre_election: true
I20260928 15:55:48.965384 10852 leader_election.cc:302] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb0a5f69762425eb37722733fdfc091; no voters: 3bca4275847a40f698698d5721002566, e28189d4fd3742469da4651a10196f57
I20260928 15:55:48.965821 11249 raft_consensus.cc:2726] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 15:55:49.091423 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.085165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.22.157.194:39431)
0928 15:55:49.085387 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:49.085391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:49.085882 (+ 491us) server_negotiation.cc:407] Connection header received
0928 15:55:49.086078 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:49.086083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:49.086166 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:49.086305 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:55:49.086773 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:49.087588 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:49.089215 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:49.089646 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:49.089759 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:49.090292 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:55:49.090849 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:55:49.091026 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:49.091066 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 15:55:49.091135 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:55:49.099761 11177 raft_consensus.cc:1261] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 1 FOLLOWER]: Refusing update from remote peer e28189d4fd3742469da4651a10196f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:49.099818 10916 raft_consensus.cc:1261] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 1 FOLLOWER]: Refusing update from remote peer e28189d4fd3742469da4651a10196f57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:55:49.100677 11260 consensus_queue.cc:1035] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bca4275847a40f698698d5721002566" member_type: VOTER last_known_addr { host: "127.22.157.195" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:49.101182 11248 consensus_queue.cc:1035] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb0a5f69762425eb37722733fdfc091" member_type: VOTER last_known_addr { host: "127.22.157.193" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:55:49.130307 10773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:49.130584 10773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 15:55:49.134706 10773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 626fad28290b4729be2819020012d0ae: Sending DeleteTablet for 3 replicas of tablet 32f26e387271420fa857646aef3d9773
I20260928 15:55:49.135954 10896 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f26e387271420fa857646aef3d9773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:49 UTC) from {username='slave'} at 127.0.0.1:60262
I20260928 15:55:49.136003 11157 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f26e387271420fa857646aef3d9773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:49 UTC) from {username='slave'} at 127.0.0.1:58090
I20260928 15:55:49.136242 11027 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f26e387271420fa857646aef3d9773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 15:55:49 UTC) from {username='slave'} at 127.0.0.1:52642
I20260928 15:55:49.139484 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 10834
I20260928 15:55:49.147293 11280 tablet_replica.cc:330] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: stopping tablet replica
I20260928 15:55:49.148054 11280 raft_consensus.cc:2229] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 1 LEADER]: Raft consensus shutting down.
I20260928 15:55:49.149087 11280 raft_consensus.cc:2258] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:49.150382 11278 tablet_replica.cc:330] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091: stopping tablet replica
I20260928 15:55:49.151046 11279 tablet_replica.cc:330] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: stopping tablet replica
I20260928 15:55:49.151450 11279 raft_consensus.cc:2229] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:49.151923 11279 raft_consensus.cc:2258] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:49.154103 11278 raft_consensus.cc:2229] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 15:55:49.154371 11280 ts_tablet_manager.cc:1905] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:49.154661 11278 raft_consensus.cc:2258] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 15:55:49.157346 11279 ts_tablet_manager.cc:1905] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:49.159183 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.140810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.22.157.193:42023)
0928 15:55:49.141117 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:49.141127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:49.148247 (+ 7120us) server_negotiation.cc:407] Connection header received
0928 15:55:49.149231 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:49.149237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:49.149369 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:49.149478 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:49.150517 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:49.151699 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:49.153693 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:49.154409 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:49.154568 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:49.157291 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:49.157304 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:49.157321 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:49.158097 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:49.158168 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:49.158199 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:49.158306 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:49.158692 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 15:55:49.158819 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:55:49.160635 10956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:49.162231 11280 ts_tablet_manager.cc:1918] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:49.162439 11280 log.cc:1196] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-1/wal/wals/32f26e387271420fa857646aef3d9773
I20260928 15:55:49.163067 11280 ts_tablet_manager.cc:1939] T 32f26e387271420fa857646aef3d9773 P e28189d4fd3742469da4651a10196f57: Deleting consensus metadata
I20260928 15:55:49.164774 10761 catalog_manager.cc:4841] TS e28189d4fd3742469da4651a10196f57 (127.22.157.194:39431): tablet 32f26e387271420fa857646aef3d9773 (table test-table [id=634e62dc50f7435292fbcd5e0742d627]) successfully deleted
I20260928 15:55:49.168417 11279 ts_tablet_manager.cc:1918] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 15:55:49.168629 11279 log.cc:1196] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: Deleting WAL directory at /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790610618730370-23159-0/minicluster-data/ts-2/wal/wals/32f26e387271420fa857646aef3d9773
I20260928 15:55:49.169291 11279 ts_tablet_manager.cc:1939] T 32f26e387271420fa857646aef3d9773 P 3bca4275847a40f698698d5721002566: Deleting consensus metadata
I20260928 15:55:49.171746 10760 catalog_manager.cc:4841] TS 3bca4275847a40f698698d5721002566 (127.22.157.195:43833): tablet 32f26e387271420fa857646aef3d9773 (table test-table [id=634e62dc50f7435292fbcd5e0742d627]) successfully deleted
I20260928 15:55:49.161576 11278 ts_tablet_manager.cc:1905] T 32f26e387271420fa857646aef3d9773 P abb0a5f69762425eb37722733fdfc091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 15:55:49.263362 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 10834
W20260928 15:55:49.289624 10760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.157.193:42023 (error 108)
I20260928 15:55:49.289841 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 10965
I20260928 15:55:49.292205 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.291263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.0.0.1:60284)
0928 15:55:49.291815 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:49.291951 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":290,"threads_started":1}
W20260928 15:55:49.292593 10760 catalog_manager.cc:4584] TS abb0a5f69762425eb37722733fdfc091 (127.22.157.193:42023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32f26e387271420fa857646aef3d9773: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42023: connect: Connection refused (error 111)
I20260928 15:55:49.298519 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.290648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.22.157.194:39431)
0928 15:55:49.290848 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:49.290855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:49.290997 (+ 142us) server_negotiation.cc:407] Connection header received
0928 15:55:49.291225 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:49.291231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:49.291341 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:49.291449 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:49.292174 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:49.293366 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:49.294665 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:49.295402 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:49.295632 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:49.296646 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:49.296658 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:49.296684 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:49.297684 (+ 1000us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:49.297743 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:49.297768 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:49.297884 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:49.298062 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 15:55:49.298172 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:55:49.299292 11087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:49.346328 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.345858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.0.0.1:60292)
0928 15:55:49.346094 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:49.346182 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 15:55:49.347445 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.347062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.0.0.1:60294)
0928 15:55:49.347241 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:49.347308 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 15:55:49.407912 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.407376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.0.0.1:60302)
0928 15:55:49.407656 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:49.407762 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:55:49.409024 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.408602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.0.0.1:60312)
0928 15:55:49.408783 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:49.408878 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 15:55:49.416416 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 10965
I20260928 15:55:49.436461 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11095
I20260928 15:55:49.443791 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.437343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.22.157.195:43833)
0928 15:55:49.437533 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:49.437582 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:49.437607 (+ 25us) server_negotiation.cc:407] Connection header received
0928 15:55:49.437787 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:49.437791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:49.437877 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:49.438002 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:49.438417 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:49.439164 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:49.440486 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:49.441064 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:49.441207 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:49.442365 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:49.442375 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:49.442390 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:49.443010 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:49.443064 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:49.443085 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:49.443226 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:49.443359 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 15:55:49.443466 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 15:55:49.444566 11217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:49.489817 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.489238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.0.0.1:60318)
0928 15:55:49.489501 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:49.489627 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 15:55:49.491011 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:49.490523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.193:42023 (local address 127.0.0.1:60322)
0928 15:55:49.490751 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 15:55:49.490839 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.193:42023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 15:55:49.536311 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 11095
I20260928 15:55:49.553897 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 10743
I20260928 15:55:49.555029 10804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:49.666819 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 10743
2026-09-28T15:55:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3529 ms)
[----------] 6 tests from EncryptionPolicyTest (23210 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:55:49 dist-test-slave-1vw0 krb5kdc[11290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:55:49 dist-test-slave-1vw0 krb5kdc[11290](info): set up 2 sockets
Sep 28 15:55:49 dist-test-slave-1vw0 krb5kdc[11290](info): commencing operation
krb5kdc: starting...
W20260928 15:55:50.805429 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:55:50 dist-test-slave-1vw0 krb5kdc[11290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:55:51.109203 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:51.452981 11303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:51.453297 11303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:51.453482 11303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:51.461165 11303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:51.461356 11303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:51.461457 11303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:51.461630 11303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:51.461722 11303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:51.472362 11303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:51.473848 11303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:51.475350 11303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:51.489022 11309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:51.489120 11308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:51.489110 11311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:51.490577 11303 server_base.cc:1034] running on GCE node
I20260928 15:55:51.491542 11303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:51.493238 11303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:51.494539 11303 hybrid_clock.cc:648] HybridClock initialized: now 1790610951494464 us; error 65 us; skew 500 ppm
I20260928 15:55:51.494995 11303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:51 dist-test-slave-1vw0 krb5kdc[11290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:51.498780 11303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:55:51.500505 11303 webserver.cc:460] Webserver started at http://127.0.0.1:35769/ using document root <none> and password file <none>
I20260928 15:55:51.501361 11303 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:51.501544 11303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:51.501873 11303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:51.505213 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "879e6b1bdeea4258bf2636f7241fd5f9"
format_stamp: "Formatted at 2026-09-28 15:55:51 on dist-test-slave-1vw0"
server_key: "8194f04a0cb3942b71b785152e9dfd95"
server_key_iv: "d2fc5dee7c9b1436d86138203baef4cf"
server_key_version: "encryptionkey@0"
I20260928 15:55:51.506054 11303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "879e6b1bdeea4258bf2636f7241fd5f9"
format_stamp: "Formatted at 2026-09-28 15:55:51 on dist-test-slave-1vw0"
server_key: "8194f04a0cb3942b71b785152e9dfd95"
server_key_iv: "d2fc5dee7c9b1436d86138203baef4cf"
server_key_version: "encryptionkey@0"
I20260928 15:55:51.512306 11303 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 15:55:51.516372 11318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:51.518096 11303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:55:51.518304 11303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "879e6b1bdeea4258bf2636f7241fd5f9"
format_stamp: "Formatted at 2026-09-28 15:55:51 on dist-test-slave-1vw0"
server_key: "8194f04a0cb3942b71b785152e9dfd95"
server_key_iv: "d2fc5dee7c9b1436d86138203baef4cf"
server_key_version: "encryptionkey@0"
I20260928 15:55:51.518532 11303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:51.538326 11303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:51.543234 11303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:51.543589 11303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:51.563910 11303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32775
I20260928 15:55:51.564018 11369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32775 every 8 connection(s)
I20260928 15:55:51.565445 11303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:51.568077 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 11303
I20260928 15:55:51.568306 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:51.569209 23159 external_mini_cluster.cc:1428] Setting key abbeda602699be015b9daf3f04b7d7bf
I20260928 15:55:51.570138 11370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:55:51 dist-test-slave-1vw0 krb5kdc[11290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:55:51.586545 11370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9: Bootstrap starting.
I20260928 15:55:51.590540 11370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:51.591785 11370 log.cc:824] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:51.595405 11370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9: No bootstrap required, opened a new log
I20260928 15:55:51.595688 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:51.571327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53271 (local address 127.0.0.1:32775)
0928 15:55:51.572167 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:51.572179 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:51.572220 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:55:51.573894 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:51.573919 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:51.574400 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:51.574781 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:51.576773 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:51.578239 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:51.580126 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:51.580941 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:51.581117 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:51.586596 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:51.586605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:51.586622 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:51.586663 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:51.590430 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:51.591529 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:51.591540 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:51.591552 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:51.591799 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:51.592366 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:51.592374 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:51.592379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:51.593082 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:51.593423 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:51.593899 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 15:55:51.594287 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":388,"threads_started":1}
I20260928 15:55:51.600028 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:55:51.604234 11370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879e6b1bdeea4258bf2636f7241fd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260928 15:55:51.604801 11370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:51.604949 11370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879e6b1bdeea4258bf2636f7241fd5f9, State: Initialized, Role: FOLLOWER
I20260928 15:55:51.605718 11370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879e6b1bdeea4258bf2636f7241fd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260928 15:55:51.606034 11370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:51.606211 11370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:51.606422 11370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:51.609401 11370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879e6b1bdeea4258bf2636f7241fd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260928 15:55:51.609939 11370 leader_election.cc:302] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 879e6b1bdeea4258bf2636f7241fd5f9; no voters:
I20260928 15:55:51.610586 11370 leader_election.cc:288] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:51.610731 11380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:51.612461 11380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 879e6b1bdeea4258bf2636f7241fd5f9, State: Running, Role: LEADER
I20260928 15:55:51.613125 11380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879e6b1bdeea4258bf2636f7241fd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260928 15:55:51.613431 11370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:51.617520 11382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 879e6b1bdeea4258bf2636f7241fd5f9. Latest consensus state: current_term: 1 leader_uuid: "879e6b1bdeea4258bf2636f7241fd5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879e6b1bdeea4258bf2636f7241fd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } } }
I20260928 15:55:51.618013 11382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:51.619737 11381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "879e6b1bdeea4258bf2636f7241fd5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "879e6b1bdeea4258bf2636f7241fd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } } }
I20260928 15:55:51.620182 11381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:51.622248 11389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:51.623515 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:51.605030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44007 (local address 127.0.0.1:32775)
0928 15:55:51.605228 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:51.605233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:51.605934 (+ 701us) server_negotiation.cc:407] Connection header received
0928 15:55:51.606138 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:51.606144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:51.606236 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:51.606348 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:51.607868 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:51.609098 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:51.610484 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:51.611368 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:51.611519 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:51.614541 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:51.614548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:51.614552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:51.614594 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:51.619201 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:51.620808 (+ 1607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:51.620820 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:51.620826 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:51.621114 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:51.621777 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:51.621781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:51.621782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:51.622156 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:51.622526 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:51.622866 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 15:55:51.623080 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:55:51.628140 11389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:51.638585 11389 catalog_manager.cc:1347] Generated new cluster ID: 67fa1c4539fb4e8dae471fa48ad16919
I20260928 15:55:51.638739 11389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:51.653398 11389 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:51.655387 11389 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:51.672255 11389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 879e6b1bdeea4258bf2636f7241fd5f9: Generated new TSK 0
I20260928 15:55:51.673475 11389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:55:51.684007 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11303
I20260928 15:55:51.685411 11365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:51.784740 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 11303
W20260928 15:55:51.806761 11288 connection.cc:535] client connection to 127.0.0.1:32775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32775): Connection reset by peer (error 104)
2026-09-28T15:55:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2112 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:55:51 dist-test-slave-1vw0 krb5kdc[11402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:55:51 dist-test-slave-1vw0 krb5kdc[11402](info): set up 2 sockets
Sep 28 15:55:51 dist-test-slave-1vw0 krb5kdc[11402](info): commencing operation
krb5kdc: starting...
W20260928 15:55:52.945560 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:55:53 dist-test-slave-1vw0 krb5kdc[11402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:55:53.244673 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:53.591368 11415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:53.591706 11415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:53.591871 11415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:53.599514 11415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:53.599659 11415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:53.599754 11415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:53.599874 11415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:53.599956 11415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:53.610545 11415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:53.612038 11415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:53.613513 11415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:53.626879 11421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:53.626847 11420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:53.626847 11423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:53.627763 11415 server_base.cc:1034] running on GCE node
I20260928 15:55:53.628588 11415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:53.630241 11415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:53.631493 11415 hybrid_clock.cc:648] HybridClock initialized: now 1790610953631436 us; error 56 us; skew 500 ppm
I20260928 15:55:53.631860 11415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:53 dist-test-slave-1vw0 krb5kdc[11402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:53.635439 11415 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:55:53.637303 11415 webserver.cc:460] Webserver started at http://127.0.0.1:33211/ using document root <none> and password file <none>
I20260928 15:55:53.638100 11415 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:53.638293 11415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:53.638687 11415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:53.642277 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "4730bca0f5d64f7d86decd3e47c10352"
format_stamp: "Formatted at 2026-09-28 15:55:53 on dist-test-slave-1vw0"
server_key: "f9616ecb12cc3143210846fa3874d8d2"
server_key_iv: "0f55dc417855326faa0e99ab8b8755ff"
server_key_version: "encryptionkey@0"
I20260928 15:55:53.643267 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "4730bca0f5d64f7d86decd3e47c10352"
format_stamp: "Formatted at 2026-09-28 15:55:53 on dist-test-slave-1vw0"
server_key: "f9616ecb12cc3143210846fa3874d8d2"
server_key_iv: "0f55dc417855326faa0e99ab8b8755ff"
server_key_version: "encryptionkey@0"
I20260928 15:55:53.649595 11415 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:55:53.653652 11430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:53.655138 11415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260928 15:55:53.655346 11415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "4730bca0f5d64f7d86decd3e47c10352"
format_stamp: "Formatted at 2026-09-28 15:55:53 on dist-test-slave-1vw0"
server_key: "f9616ecb12cc3143210846fa3874d8d2"
server_key_iv: "0f55dc417855326faa0e99ab8b8755ff"
server_key_version: "encryptionkey@0"
I20260928 15:55:53.655623 11415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:53.683001 11415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:53.688869 11415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:53.689225 11415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:53.709434 11415 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37811
I20260928 15:55:53.709471 11481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37811 every 8 connection(s)
I20260928 15:55:53.710971 11415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:53.714679 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 11415
I20260928 15:55:53.714876 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:53.715315 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:53.715353 23159 external_mini_cluster.cc:1428] Setting key d34b44e138e61b690b226cd0125ef2f8
I20260928 15:55:53.729543 11482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352: Bootstrap starting.
I20260928 15:55:53.731351 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:53.717331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55939 (local address 127.0.0.1:37811)
0928 15:55:53.718286 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:53.718306 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:53.718652 (+ 346us) server_negotiation.cc:407] Connection header received
0928 15:55:53.720939 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:53.720964 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:53.721493 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:53.722012 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:53.722673 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:53.724126 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:53.725847 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:53.726772 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:53.727041 (+ 269us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:53.728594 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:53.728606 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:53.728622 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:53.729338 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:53.729451 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:53.729481 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:53.729682 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:53.730040 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 15:55:53.730249 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":419,"threads_started":1}
I20260928 15:55:53.733302 11482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:53.734728 11482 log.cc:824] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:53.736240 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:55:53.737962 11482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352: No bootstrap required, opened a new log
I20260928 15:55:53.746214 11482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730bca0f5d64f7d86decd3e47c10352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37811 } }
I20260928 15:55:53.746677 11482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:53.746803 11482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4730bca0f5d64f7d86decd3e47c10352, State: Initialized, Role: FOLLOWER
I20260928 15:55:53.747563 11482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730bca0f5d64f7d86decd3e47c10352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37811 } }
I20260928 15:55:53.747828 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:53.741088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54537 (local address 127.0.0.1:37811)
0928 15:55:53.741335 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:53.741343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:53.741671 (+ 328us) server_negotiation.cc:407] Connection header received
0928 15:55:53.741915 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:53.741919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:53.741987 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:53.742105 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:53.742640 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:53.743460 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:53.744931 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:53.745407 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:53.745577 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:53.746831 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:53.746838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:53.746845 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:53.746869 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:53.746944 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:53.746965 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:53.747106 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:53.747377 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 15:55:53.747438 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:55:53.747918 11482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:53.748076 11482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:53.748278 11482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:53.750900 11482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730bca0f5d64f7d86decd3e47c10352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37811 } }
I20260928 15:55:53.751380 11482 leader_election.cc:302] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4730bca0f5d64f7d86decd3e47c10352; no voters:
I20260928 15:55:53.751968 11482 leader_election.cc:288] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:53.752130 11492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:53.752938 11492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [term 1 LEADER]: Becoming Leader. State: Replica: 4730bca0f5d64f7d86decd3e47c10352, State: Running, Role: LEADER
I20260928 15:55:53.753487 11492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730bca0f5d64f7d86decd3e47c10352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37811 } }
I20260928 15:55:53.754370 11482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:53.757135 11493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4730bca0f5d64f7d86decd3e47c10352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730bca0f5d64f7d86decd3e47c10352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37811 } } }
I20260928 15:55:53.757071 11494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4730bca0f5d64f7d86decd3e47c10352. Latest consensus state: current_term: 1 leader_uuid: "4730bca0f5d64f7d86decd3e47c10352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730bca0f5d64f7d86decd3e47c10352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37811 } } }
I20260928 15:55:53.757617 11494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:53.760862 11493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:53.761879 11497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:53.769344 11497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:53.780746 11497 catalog_manager.cc:1347] Generated new cluster ID: 6fda534e80924f3cbcdb0f26cbdf3729
I20260928 15:55:53.780903 11497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:53.797633 11497 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:53.799716 11497 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:53.811055 11497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4730bca0f5d64f7d86decd3e47c10352: Generated new TSK 0
I20260928 15:55:53.811928 11497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:55:53.819392 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11415
W20260928 15:55:53.820670 11477 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55939
W20260928 15:55:53.821601 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11415: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:55:53.821813 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 11415
W20260928 15:55:53.837589 11400 connection.cc:535] client connection to 127.0.0.1:37811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37811): Connection reset by peer (error 104)
2026-09-28T15:55:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:55:53 dist-test-slave-1vw0 krb5kdc[11513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:55:53 dist-test-slave-1vw0 krb5kdc[11513](info): set up 2 sockets
Sep 28 15:55:53 dist-test-slave-1vw0 krb5kdc[11513](info): commencing operation
krb5kdc: starting...
W20260928 15:55:54.941551 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:55:55 dist-test-slave-1vw0 krb5kdc[11513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:55:55.240247 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:55.587314 11526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:55.587651 11526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:55.587828 11526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:55.595443 11526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:55.595602 11526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:55.595706 11526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:55.595788 11526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:55.595861 11526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:55.606472 11526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46493
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:55.607936 11526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:55.609395 11526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:55.623031 11531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:55.622980 11532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:55.622993 11534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:55.623901 11526 server_base.cc:1034] running on GCE node
I20260928 15:55:55.624922 11526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:55.626579 11526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:55.627827 11526 hybrid_clock.cc:648] HybridClock initialized: now 1790610955627783 us; error 41 us; skew 500 ppm
I20260928 15:55:55.628243 11526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:55 dist-test-slave-1vw0 krb5kdc[11513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:55.631573 11526 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:55:55.633337 11526 webserver.cc:460] Webserver started at http://127.0.0.1:39175/ using document root <none> and password file <none>
I20260928 15:55:55.634099 11526 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:55.634289 11526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:55.634608 11526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:55.637889 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "1c4a9c3d333d45c09d6c2b95ce98064e"
format_stamp: "Formatted at 2026-09-28 15:55:55 on dist-test-slave-1vw0"
server_key: "7be3d8037fe3f0361b62974ebb0a2a63"
server_key_iv: "7dabb65ad5949f02c62e49099e6c4b05"
server_key_version: "encryptionkey@0"
I20260928 15:55:55.638712 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "1c4a9c3d333d45c09d6c2b95ce98064e"
format_stamp: "Formatted at 2026-09-28 15:55:55 on dist-test-slave-1vw0"
server_key: "7be3d8037fe3f0361b62974ebb0a2a63"
server_key_iv: "7dabb65ad5949f02c62e49099e6c4b05"
server_key_version: "encryptionkey@0"
I20260928 15:55:55.644359 11526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 15:55:55.648090 11541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:55.649655 11526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 15:55:55.649855 11526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "1c4a9c3d333d45c09d6c2b95ce98064e"
format_stamp: "Formatted at 2026-09-28 15:55:55 on dist-test-slave-1vw0"
server_key: "7be3d8037fe3f0361b62974ebb0a2a63"
server_key_iv: "7dabb65ad5949f02c62e49099e6c4b05"
server_key_version: "encryptionkey@0"
I20260928 15:55:55.650085 11526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:55.656278 11526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:55.660928 11526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:55.661255 11526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:55.681747 11526 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46493
I20260928 15:55:55.681770 11592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46493 every 8 connection(s)
I20260928 15:55:55.683270 11526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:55.687374 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:55.690361 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 11526
I20260928 15:55:55.690568 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:55.691144 23159 external_mini_cluster.cc:1428] Setting key 51c9f22955c9da1c3148bd6491200049
I20260928 15:55:55.700356 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:55.693350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52711 (local address 127.0.0.1:46493)
0928 15:55:55.694244 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:55.694258 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:55.694307 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:55:55.696225 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:55.696251 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:55.696669 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:55.697210 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:55.698488 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:55.698495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:55.698510 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:55.698527 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:55.698594 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:55.698615 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:55.698892 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:55.699074 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 15:55:55.699280 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":359,"threads_started":1}
I20260928 15:55:55.700471 11593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e: Bootstrap starting.
I20260928 15:55:55.704092 11593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:55.704679 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:55:55.705422 11593 log.cc:824] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:55.709112 11593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e: No bootstrap required, opened a new log
I20260928 15:55:55.713425 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:55.710334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37975 (local address 127.0.0.1:46493)
0928 15:55:55.710795 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:55.710801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:55.710972 (+ 171us) server_negotiation.cc:407] Connection header received
0928 15:55:55.711229 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:55.711233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:55.711289 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:55.711435 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:55.712580 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:55.712585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:55.712588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:55.712607 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:55.712655 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:55.712673 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:55.712907 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:55.713034 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 15:55:55.713108 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260928 15:55:55.718468 11593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4a9c3d333d45c09d6c2b95ce98064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260928 15:55:55.719039 11593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:55.719156 11593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c4a9c3d333d45c09d6c2b95ce98064e, State: Initialized, Role: FOLLOWER
I20260928 15:55:55.719862 11593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4a9c3d333d45c09d6c2b95ce98064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260928 15:55:55.720079 11593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:55.720183 11593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:55.720312 11593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:55.722611 11593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4a9c3d333d45c09d6c2b95ce98064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260928 15:55:55.723172 11593 leader_election.cc:302] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c4a9c3d333d45c09d6c2b95ce98064e; no voters:
I20260928 15:55:55.723920 11593 leader_election.cc:288] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:55.724128 11603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:55.725013 11603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [term 1 LEADER]: Becoming Leader. State: Replica: 1c4a9c3d333d45c09d6c2b95ce98064e, State: Running, Role: LEADER
I20260928 15:55:55.725831 11603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4a9c3d333d45c09d6c2b95ce98064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260928 15:55:55.726125 11593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:55.729025 11605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c4a9c3d333d45c09d6c2b95ce98064e. Latest consensus state: current_term: 1 leader_uuid: "1c4a9c3d333d45c09d6c2b95ce98064e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4a9c3d333d45c09d6c2b95ce98064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } } }
I20260928 15:55:55.729341 11604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c4a9c3d333d45c09d6c2b95ce98064e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4a9c3d333d45c09d6c2b95ce98064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } } }
I20260928 15:55:55.729588 11605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:55.729637 11604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:55.734043 11609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:55.739375 11609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:55.749698 11609 catalog_manager.cc:1347] Generated new cluster ID: fe049b2619904276b7fc332ec87380f9
I20260928 15:55:55.749833 11609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:55.774479 11609 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:55.775048 11609 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:55.788151 11609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c4a9c3d333d45c09d6c2b95ce98064e: Generated new TSK 0
I20260928 15:55:55.789460 11609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:55:55.805532 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11526
W20260928 15:55:55.807003 11588 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52711
W20260928 15:55:55.807888 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11526: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:55:55.808092 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 11526
2026-09-28T15:55:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:55:55 dist-test-slave-1vw0 krb5kdc[11624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:55:55 dist-test-slave-1vw0 krb5kdc[11624](info): set up 2 sockets
Sep 28 15:55:55 dist-test-slave-1vw0 krb5kdc[11624](info): commencing operation
krb5kdc: starting...
W20260928 15:55:56.937376 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:55:57 dist-test-slave-1vw0 krb5kdc[11624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:55:57.238410 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:57.590698 11637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:57.591089 11637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:57.591259 11637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:57.598920 11637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:57.599267 11637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:57.599399 11637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:57.599519 11637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:57.599607 11637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:57.610740 11637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:57.612246 11637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:57.613767 11637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:57.627271 11642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:57.627326 11645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:57.627303 11643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:57.628286 11637 server_base.cc:1034] running on GCE node
I20260928 15:55:57.629253 11637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:57.630980 11637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:57.632234 11637 hybrid_clock.cc:648] HybridClock initialized: now 1790610957632188 us; error 47 us; skew 500 ppm
I20260928 15:55:57.632637 11637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:57 dist-test-slave-1vw0 krb5kdc[11624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:57.642817 11637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:55:57.644709 11637 webserver.cc:460] Webserver started at http://127.0.0.1:45959/ using document root <none> and password file <none>
I20260928 15:55:57.645525 11637 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:57.645646 11637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:57.645922 11637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:57.649408 11637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "c507afae15b2448e8763aa30fbcd2678"
format_stamp: "Formatted at 2026-09-28 15:55:57 on dist-test-slave-1vw0"
server_key: "d3953c210d38f71d3e695de49ebad734"
server_key_iv: "5c258c67355f1ea29c9bdfebfb8085d0"
server_key_version: "encryptionkey@0"
I20260928 15:55:57.650220 11637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "c507afae15b2448e8763aa30fbcd2678"
format_stamp: "Formatted at 2026-09-28 15:55:57 on dist-test-slave-1vw0"
server_key: "d3953c210d38f71d3e695de49ebad734"
server_key_iv: "5c258c67355f1ea29c9bdfebfb8085d0"
server_key_version: "encryptionkey@0"
I20260928 15:55:57.656711 11637 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 15:55:57.661306 11652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:57.662935 11637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:55:57.663190 11637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "c507afae15b2448e8763aa30fbcd2678"
format_stamp: "Formatted at 2026-09-28 15:55:57 on dist-test-slave-1vw0"
server_key: "d3953c210d38f71d3e695de49ebad734"
server_key_iv: "5c258c67355f1ea29c9bdfebfb8085d0"
server_key_version: "encryptionkey@0"
I20260928 15:55:57.663440 11637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:57.676577 11637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:57.681852 11637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:57.682193 11637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:57.706624 11637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32987
I20260928 15:55:57.706664 11703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32987 every 8 connection(s)
I20260928 15:55:57.708710 11637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:57.709365 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 11637
I20260928 15:55:57.709563 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:57.710204 23159 external_mini_cluster.cc:1428] Setting key f9bf160b2712dd37144377ceb490fd1e
I20260928 15:55:57.714406 11704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 15:55:57 dist-test-slave-1vw0 krb5kdc[11624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:55:57.730947 11704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678: Bootstrap starting.
I20260928 15:55:57.736156 11704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:57.737876 11704 log.cc:824] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:57.742434 11704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678: No bootstrap required, opened a new log
I20260928 15:55:57.744065 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:57.712445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47727 (local address 127.0.0.1:32987)
0928 15:55:57.713539 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:57.713556 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:57.713610 (+ 54us) server_negotiation.cc:407] Connection header received
0928 15:55:57.716056 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:57.716088 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:57.716644 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:57.717178 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:57.719625 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:57.721913 (+ 2288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:57.723963 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:57.725515 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:57.725791 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:57.731540 (+ 5749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:57.731554 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:57.731577 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:57.731656 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:57.736664 (+ 5008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:57.737992 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:57.738003 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:57.738014 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:57.738243 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:57.738885 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:57.738891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:57.738895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:57.739843 (+ 948us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:57.741677 (+ 1834us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:57.742024 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 15:55:57.742580 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":801,"thread_start_us":428,"threads_started":1}
I20260928 15:55:57.749487 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:55:57.752559 11704 raft_consensus.cc:348] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c507afae15b2448e8763aa30fbcd2678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20260928 15:55:57.753100 11704 raft_consensus.cc:374] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:57.753223 11704 raft_consensus.cc:729] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c507afae15b2448e8763aa30fbcd2678, State: Initialized, Role: FOLLOWER
I20260928 15:55:57.753854 11704 consensus_queue.cc:260] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c507afae15b2448e8763aa30fbcd2678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20260928 15:55:57.754117 11704 raft_consensus.cc:388] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:57.754347 11704 raft_consensus.cc:482] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:57.754619 11704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:57.758200 11704 raft_consensus.cc:504] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c507afae15b2448e8763aa30fbcd2678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20260928 15:55:57.758841 11704 leader_election.cc:302] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c507afae15b2448e8763aa30fbcd2678; no voters:
I20260928 15:55:57.759580 11704 leader_election.cc:288] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:57.759718 11714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:57.762238 11704 sys_catalog.cc:564] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:57.766546 11714 raft_consensus.cc:686] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [term 1 LEADER]: Becoming Leader. State: Replica: c507afae15b2448e8763aa30fbcd2678, State: Running, Role: LEADER
I20260928 15:55:57.767525 11714 consensus_queue.cc:237] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c507afae15b2448e8763aa30fbcd2678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } }
I20260928 15:55:57.771464 11716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c507afae15b2448e8763aa30fbcd2678. Latest consensus state: current_term: 1 leader_uuid: "c507afae15b2448e8763aa30fbcd2678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c507afae15b2448e8763aa30fbcd2678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } } }
I20260928 15:55:57.772040 11716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:57.771430 11715 sys_catalog.cc:455] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c507afae15b2448e8763aa30fbcd2678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c507afae15b2448e8763aa30fbcd2678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32987 } } }
I20260928 15:55:57.772746 11715 sys_catalog.cc:458] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:57.773767 11725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:57.775134 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:57.754380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53335 (local address 127.0.0.1:32987)
0928 15:55:57.754630 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:57.754639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:57.755125 (+ 486us) server_negotiation.cc:407] Connection header received
0928 15:55:57.755316 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:57.755319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:57.755387 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:57.755479 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:57.757680 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:57.758616 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:57.760818 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:57.761514 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:57.761627 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:57.765482 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:57.765490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:57.765498 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:55:57.765559 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:57.770403 (+ 4844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:57.771781 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:57.771790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:57.771796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:57.772065 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:55:57.772759 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:55:57.772766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:55:57.772769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:55:57.773257 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:57.773595 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:57.774435 (+ 840us) server_negotiation.cc:299] Negotiation successful
0928 15:55:57.774700 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:55:57.783773 11725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:57.796254 11725 catalog_manager.cc:1347] Generated new cluster ID: 003e43fdcb624d7e904185ec0810e665
I20260928 15:55:57.796396 11725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:57.814582 11725 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:57.816064 11725 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:57.829830 11725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c507afae15b2448e8763aa30fbcd2678: Generated new TSK 0
I20260928 15:55:57.830842 11725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:55:57.844560 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11637
I20260928 15:55:57.846031 11699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:55:57.941097 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 11637
2026-09-28T15:55:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2129 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:55:58 dist-test-slave-1vw0 krb5kdc[11736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:55:58 dist-test-slave-1vw0 krb5kdc[11736](info): set up 2 sockets
Sep 28 15:55:58 dist-test-slave-1vw0 krb5kdc[11736](info): commencing operation
krb5kdc: starting...
W20260928 15:55:59.105711 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.002s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:55:59 dist-test-slave-1vw0 krb5kdc[11736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:55:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:55:59.411990 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:55:59.765683 11749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:55:59.766064 11749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:55:59.766224 11749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:55:59.774008 11749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:55:59.774199 11749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:55:59.774304 11749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:55:59.774410 11749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:55:59.774488 11749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:55:59.790654 11749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:55:59.792217 11749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:55:59.793689 11749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:55:59.808295 11754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:59.810866 11749 server_base.cc:1034] running on GCE node
W20260928 15:55:59.810318 11755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:55:59.811200 11757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:55:59.812386 11749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:55:59.814549 11749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:55:59.815825 11749 hybrid_clock.cc:648] HybridClock initialized: now 1790610959815747 us; error 56 us; skew 500 ppm
I20260928 15:55:59.816335 11749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:55:59 dist-test-slave-1vw0 krb5kdc[11736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:55:59.820755 11749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:55:59.822449 11749 webserver.cc:460] Webserver started at http://127.0.0.1:43777/ using document root <none> and password file <none>
I20260928 15:55:59.823182 11749 fs_manager.cc:359] Metadata directory not provided
I20260928 15:55:59.823318 11749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:55:59.823665 11749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:55:59.826977 11749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "17aff2c213ac453997e7389b3d1d30a7"
format_stamp: "Formatted at 2026-09-28 15:55:59 on dist-test-slave-1vw0"
server_key: "50dfb3f6329767cb2d7a782f5ff88dca"
server_key_iv: "7468aaf7fd0dd24b247cb0c2706eb7fb"
server_key_version: "encryptionkey@0"
I20260928 15:55:59.827816 11749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "17aff2c213ac453997e7389b3d1d30a7"
format_stamp: "Formatted at 2026-09-28 15:55:59 on dist-test-slave-1vw0"
server_key: "50dfb3f6329767cb2d7a782f5ff88dca"
server_key_iv: "7468aaf7fd0dd24b247cb0c2706eb7fb"
server_key_version: "encryptionkey@0"
I20260928 15:55:59.833838 11749 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:55:59.837671 11764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:55:59.839267 11749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:55:59.839483 11749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "17aff2c213ac453997e7389b3d1d30a7"
format_stamp: "Formatted at 2026-09-28 15:55:59 on dist-test-slave-1vw0"
server_key: "50dfb3f6329767cb2d7a782f5ff88dca"
server_key_iv: "7468aaf7fd0dd24b247cb0c2706eb7fb"
server_key_version: "encryptionkey@0"
I20260928 15:55:59.839702 11749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:55:59.860888 11749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:55:59.865846 11749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:55:59.866160 11749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:55:59.886931 11749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45431
I20260928 15:55:59.886968 11815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45431 every 8 connection(s)
I20260928 15:55:59.888371 11749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:55:59.892069 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 11749
I20260928 15:55:59.892266 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:55:59.892458 11816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:55:59.892838 23159 external_mini_cluster.cc:1428] Setting key 7af599dc18bd4de10750520575d2a7e0
I20260928 15:55:59.907115 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:59.894594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54173 (local address 127.0.0.1:45431)
0928 15:55:59.895422 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:59.895434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:59.895487 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:55:59.897440 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:59.897465 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:59.897993 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:59.898353 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:59.898986 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:59.900421 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:59.902316 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:59.903033 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:59.903178 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:59.904525 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:59.904533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:59.904547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:59.905223 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:59.905313 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:59.905339 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:59.905432 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:59.905698 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 15:55:59.906035 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":374,"threads_started":1}
I20260928 15:55:59.907290 11816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7: Bootstrap starting.
I20260928 15:55:59.911120 11816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7: Neither blocks nor log segments found. Creating new log.
I20260928 15:55:59.911722 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:55:59.912487 11816 log.cc:824] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:55:59.916262 11816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7: No bootstrap required, opened a new log
I20260928 15:55:59.924651 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:55:59.917263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38747 (local address 127.0.0.1:45431)
0928 15:55:59.917451 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 15:55:59.917457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:55:59.917899 (+ 442us) server_negotiation.cc:407] Connection header received
0928 15:55:59.918069 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:55:59.918075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:55:59.918167 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:55:59.918288 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:55:59.918834 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:59.920031 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:59.921533 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:55:59.922281 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:55:59.922424 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:55:59.923765 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:55:59.923773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:55:59.923782 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:55:59.923810 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:55:59.923868 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:55:59.923894 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:55:59.923992 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 15:55:59.924163 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 15:55:59.924265 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:55:59.926317 11816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aff2c213ac453997e7389b3d1d30a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } }
I20260928 15:55:59.926746 11816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:55:59.926863 11816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17aff2c213ac453997e7389b3d1d30a7, State: Initialized, Role: FOLLOWER
I20260928 15:55:59.927484 11816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aff2c213ac453997e7389b3d1d30a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } }
I20260928 15:55:59.927834 11816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:55:59.928002 11816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:55:59.928200 11816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:55:59.930593 11816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aff2c213ac453997e7389b3d1d30a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } }
I20260928 15:55:59.931087 11816 leader_election.cc:302] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17aff2c213ac453997e7389b3d1d30a7; no voters:
I20260928 15:55:59.931679 11816 leader_election.cc:288] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:55:59.931849 11826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:55:59.932601 11826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [term 1 LEADER]: Becoming Leader. State: Replica: 17aff2c213ac453997e7389b3d1d30a7, State: Running, Role: LEADER
I20260928 15:55:59.933266 11826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aff2c213ac453997e7389b3d1d30a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } }
I20260928 15:55:59.934120 11816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:55:59.936390 11827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17aff2c213ac453997e7389b3d1d30a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aff2c213ac453997e7389b3d1d30a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } } }
I20260928 15:55:59.936887 11827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:59.937551 11828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17aff2c213ac453997e7389b3d1d30a7. Latest consensus state: current_term: 1 leader_uuid: "17aff2c213ac453997e7389b3d1d30a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17aff2c213ac453997e7389b3d1d30a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } } }
I20260928 15:55:59.937923 11828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:55:59.938591 11833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:55:59.943259 11833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:55:59.953754 11833 catalog_manager.cc:1347] Generated new cluster ID: a6b958a404fc48908e9bc726f9a86105
I20260928 15:55:59.953884 11833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:55:59.972642 11833 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:55:59.974105 11833 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:55:59.986217 11833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17aff2c213ac453997e7389b3d1d30a7: Generated new TSK 0
I20260928 15:55:59.987116 11833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:55:59.998083 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11749
W20260928 15:55:59.999394 11811 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54173
W20260928 15:56:00.000170 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11749: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:00.000340 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 11749
2026-09-28T15:56:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2061 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:00 dist-test-slave-1vw0 krb5kdc[11847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:00 dist-test-slave-1vw0 krb5kdc[11847](info): set up 2 sockets
Sep 28 15:56:00 dist-test-slave-1vw0 krb5kdc[11847](info): commencing operation
krb5kdc: starting...
W20260928 15:56:01.129518 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:01 dist-test-slave-1vw0 krb5kdc[11847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:56:01.428527 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:01.777806 11860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:01.778209 11860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:01.778373 11860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:01.790475 11860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:01.790735 11860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:01.790843 11860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:01.790951 11860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:01.791024 11860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:01.802752 11860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:01.804152 11860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:01.805593 11860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:01.818557 11865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:01.818586 11868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:01.818642 11866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:01.819730 11860 server_base.cc:1034] running on GCE node
I20260928 15:56:01.820633 11860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:01.822252 11860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:01.823503 11860 hybrid_clock.cc:648] HybridClock initialized: now 1790610961823447 us; error 44 us; skew 500 ppm
I20260928 15:56:01.823984 11860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:01 dist-test-slave-1vw0 krb5kdc[11847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:01.827637 11860 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:56:01.829630 11860 webserver.cc:460] Webserver started at http://127.0.0.1:39029/ using document root <none> and password file <none>
I20260928 15:56:01.830411 11860 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:01.830559 11860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:01.830899 11860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:01.834223 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "3ee8f7467d3a455dbed975e5de7096a7"
format_stamp: "Formatted at 2026-09-28 15:56:01 on dist-test-slave-1vw0"
server_key: "a36bdea472801ec1e16c3d6019fc40d1"
server_key_iv: "b8e0921b583591d571871a231d1e39a7"
server_key_version: "encryptionkey@0"
I20260928 15:56:01.835034 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "3ee8f7467d3a455dbed975e5de7096a7"
format_stamp: "Formatted at 2026-09-28 15:56:01 on dist-test-slave-1vw0"
server_key: "a36bdea472801ec1e16c3d6019fc40d1"
server_key_iv: "b8e0921b583591d571871a231d1e39a7"
server_key_version: "encryptionkey@0"
I20260928 15:56:01.840634 11860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:56:01.844453 11875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:01.846143 11860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 15:56:01.846349 11860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "3ee8f7467d3a455dbed975e5de7096a7"
format_stamp: "Formatted at 2026-09-28 15:56:01 on dist-test-slave-1vw0"
server_key: "a36bdea472801ec1e16c3d6019fc40d1"
server_key_iv: "b8e0921b583591d571871a231d1e39a7"
server_key_version: "encryptionkey@0"
I20260928 15:56:01.846590 11860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:01.852875 11860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:01.857815 11860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:01.858135 11860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:01.877942 11860 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43709
I20260928 15:56:01.877974 11926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43709 every 8 connection(s)
I20260928 15:56:01.879460 11860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:01.883404 11927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:01.889561 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 11860
I20260928 15:56:01.889748 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:01.890210 23159 external_mini_cluster.cc:1428] Setting key 8941f48e58aa34ebcb46174a33d66afb
I20260928 15:56:01.895083 11927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7: Bootstrap starting.
I20260928 15:56:01.898604 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:01.892018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45351 (local address 127.0.0.1:43709)
0928 15:56:01.892941 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:01.892956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:01.893003 (+ 47us) server_negotiation.cc:407] Connection header received
0928 15:56:01.894932 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:01.894957 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:01.895380 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:01.895737 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:01.896921 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:01.896929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:01.896952 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:01.896971 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:01.897042 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:01.897064 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:01.897253 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:01.897375 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 15:56:01.897675 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":434,"threads_started":1}
I20260928 15:56:01.898813 11927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:01.900183 11927 log.cc:824] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:01.902944 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:01.903503 11927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7: No bootstrap required, opened a new log
I20260928 15:56:01.910085 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:01.907505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46391 (local address 127.0.0.1:43709)
0928 15:56:01.907721 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:01.907728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:01.908025 (+ 297us) server_negotiation.cc:407] Connection header received
0928 15:56:01.908204 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:01.908208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:01.908253 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:01.908348 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:01.909370 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:01.909374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:01.909377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:01.909396 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:01.909440 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:01.909455 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:01.909572 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:01.909686 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 15:56:01.909731 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 15:56:01.912133 11927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee8f7467d3a455dbed975e5de7096a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } }
I20260928 15:56:01.912523 11927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:01.912681 11927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee8f7467d3a455dbed975e5de7096a7, State: Initialized, Role: FOLLOWER
I20260928 15:56:01.913285 11927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee8f7467d3a455dbed975e5de7096a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } }
I20260928 15:56:01.913538 11927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:01.913686 11927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:01.913853 11927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:01.916112 11927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee8f7467d3a455dbed975e5de7096a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } }
I20260928 15:56:01.916646 11927 leader_election.cc:302] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ee8f7467d3a455dbed975e5de7096a7; no voters:
I20260928 15:56:01.917356 11927 leader_election.cc:288] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:01.917520 11937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:01.918555 11937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [term 1 LEADER]: Becoming Leader. State: Replica: 3ee8f7467d3a455dbed975e5de7096a7, State: Running, Role: LEADER
I20260928 15:56:01.919104 11937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee8f7467d3a455dbed975e5de7096a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } }
I20260928 15:56:01.919804 11927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:01.922163 11939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ee8f7467d3a455dbed975e5de7096a7. Latest consensus state: current_term: 1 leader_uuid: "3ee8f7467d3a455dbed975e5de7096a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee8f7467d3a455dbed975e5de7096a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } } }
I20260928 15:56:01.922660 11939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:01.922504 11938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ee8f7467d3a455dbed975e5de7096a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee8f7467d3a455dbed975e5de7096a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } } }
I20260928 15:56:01.922806 11938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:01.925143 11946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:01.930900 11946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:01.940465 11946 catalog_manager.cc:1347] Generated new cluster ID: c5dc01a0b8a54b6aa3c5916e174433e7
I20260928 15:56:01.940637 11946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:01.959798 11946 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:01.960289 11946 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:01.976053 11946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ee8f7467d3a455dbed975e5de7096a7: Generated new TSK 0
I20260928 15:56:01.977272 11946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:01.996399 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11860
W20260928 15:56:01.997587 11922 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45351
W20260928 15:56:01.998456 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11860: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:01.998617 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 11860
2026-09-28T15:56:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:02 dist-test-slave-1vw0 krb5kdc[11958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:02 dist-test-slave-1vw0 krb5kdc[11958](info): set up 2 sockets
Sep 28 15:56:02 dist-test-slave-1vw0 krb5kdc[11958](info): commencing operation
krb5kdc: starting...
W20260928 15:56:03.129624 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:03 dist-test-slave-1vw0 krb5kdc[11958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:56:03.432813 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:03.792865 11971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:03.793202 11971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:03.793301 11971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:03.801514 11971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:03.801652 11971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:03.801712 11971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:03.801774 11971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:03.801819 11971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:03.812131 11971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:03.813514 11971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:03.814801 11971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:03.827222 11977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:03.827239 11979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:03.828069 11976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:03.828078 11971 server_base.cc:1034] running on GCE node
I20260928 15:56:03.829084 11971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:03.830672 11971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:03.831920 11971 hybrid_clock.cc:648] HybridClock initialized: now 1790610963831895 us; error 61 us; skew 500 ppm
I20260928 15:56:03.832324 11971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:03 dist-test-slave-1vw0 krb5kdc[11958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:03.843793 11971 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:56:03.845522 11971 webserver.cc:460] Webserver started at http://127.0.0.1:40253/ using document root <none> and password file <none>
I20260928 15:56:03.846216 11971 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:03.846328 11971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:03.846590 11971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:03.849736 11971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "83646b8258664185ac373489cb9cae99"
format_stamp: "Formatted at 2026-09-28 15:56:03 on dist-test-slave-1vw0"
server_key: "9ae0e0bff6928ca8792c58c9c8cfc6ba"
server_key_iv: "780e476ca0bc310c7db17dc4126a64e1"
server_key_version: "encryptionkey@0"
I20260928 15:56:03.850502 11971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "83646b8258664185ac373489cb9cae99"
format_stamp: "Formatted at 2026-09-28 15:56:03 on dist-test-slave-1vw0"
server_key: "9ae0e0bff6928ca8792c58c9c8cfc6ba"
server_key_iv: "780e476ca0bc310c7db17dc4126a64e1"
server_key_version: "encryptionkey@0"
I20260928 15:56:03.856093 11971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:56:03.859735 11986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:03.861246 11971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:56:03.861442 11971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "83646b8258664185ac373489cb9cae99"
format_stamp: "Formatted at 2026-09-28 15:56:03 on dist-test-slave-1vw0"
server_key: "9ae0e0bff6928ca8792c58c9c8cfc6ba"
server_key_iv: "780e476ca0bc310c7db17dc4126a64e1"
server_key_version: "encryptionkey@0"
I20260928 15:56:03.861702 11971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:03.889720 11971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:03.894506 11971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:03.894833 11971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:03.915258 11971 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39321
I20260928 15:56:03.915283 12037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39321 every 8 connection(s)
I20260928 15:56:03.916821 11971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:03.921348 12038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:03.923808 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 11971
I20260928 15:56:03.924026 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:03.924454 23159 external_mini_cluster.cc:1428] Setting key b0caca95dcb8a682530672e3e2e5ec90
I20260928 15:56:03.933633 12038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99: Bootstrap starting.
I20260928 15:56:03.937047 12038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:03.938169 12038 log.cc:824] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99: Log is configured to *not* fsync() on all Append() calls
Sep 28 15:56:03 dist-test-slave-1vw0 krb5kdc[11958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:56:03.941608 12038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99: No bootstrap required, opened a new log
I20260928 15:56:03.948660 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:03.926467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:51089 (local address 127.0.0.1:39321)
0928 15:56:03.927371 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:03.927383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:03.927426 (+ 43us) server_negotiation.cc:407] Connection header received
0928 15:56:03.929279 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:03.929303 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:03.929792 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:03.930404 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:03.932004 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:03.933481 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:03.934919 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:03.935658 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:03.935862 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:03.940582 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:03.940590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:03.940605 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:03.940640 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:03.944024 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:03.945073 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:03.945080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:03.945087 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:03.945278 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:03.945853 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:03.945856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:03.945861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:03.946410 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:03.946643 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:03.947196 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 15:56:03.947544 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":406,"threads_started":1}
I20260928 15:56:03.951432 12038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83646b8258664185ac373489cb9cae99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } }
I20260928 15:56:03.951862 12038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:03.952009 12038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83646b8258664185ac373489cb9cae99, State: Initialized, Role: FOLLOWER
I20260928 15:56:03.952715 12038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83646b8258664185ac373489cb9cae99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } }
I20260928 15:56:03.952932 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:03.953029 12038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:03.953187 12038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:03.953368 12038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:03.955631 12038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83646b8258664185ac373489cb9cae99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } }
I20260928 15:56:03.956282 12038 leader_election.cc:302] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83646b8258664185ac373489cb9cae99; no voters:
I20260928 15:56:03.956974 12038 leader_election.cc:288] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:03.957166 12046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:03.959074 12038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:03.964344 12046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [term 1 LEADER]: Becoming Leader. State: Replica: 83646b8258664185ac373489cb9cae99, State: Running, Role: LEADER
I20260928 15:56:03.965150 12046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83646b8258664185ac373489cb9cae99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } }
I20260928 15:56:03.968586 12047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83646b8258664185ac373489cb9cae99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83646b8258664185ac373489cb9cae99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } } }
I20260928 15:56:03.969102 12047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:03.969719 12048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83646b8258664185ac373489cb9cae99. Latest consensus state: current_term: 1 leader_uuid: "83646b8258664185ac373489cb9cae99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83646b8258664185ac373489cb9cae99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39321 } } }
I20260928 15:56:03.970080 12048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:03.980064 12056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:03.985867 12056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:03.995976 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:03.973157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:44393 (local address 127.0.0.1:39321)
0928 15:56:03.973341 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:03.973348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:03.978050 (+ 4702us) server_negotiation.cc:407] Connection header received
0928 15:56:03.978332 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:03.978341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:03.978465 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:03.978624 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:03.980765 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:03.981953 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:03.983935 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:03.984657 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:03.984841 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:03.988170 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:03.988178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:03.988183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:03.988225 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:03.992086 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:03.993137 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:03.993144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:03.993147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:03.993293 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:03.993916 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:03.993923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:03.993927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:03.994414 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:03.994687 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:03.995399 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 15:56:03.995588 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:56:04.002055 12056 catalog_manager.cc:1347] Generated new cluster ID: 93abd4eb42e1481fa7b8b6eabbaee873
I20260928 15:56:04.002178 12056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:04.024536 12056 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:04.026607 12056 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:04.047586 12056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83646b8258664185ac373489cb9cae99: Generated new TSK 0
I20260928 15:56:04.048764 12056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:04.055223 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 11971
I20260928 15:56:04.056339 12033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:04.150048 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 11971
W20260928 15:56:04.168762 11956 connection.cc:535] client connection to 127.0.0.1:39321 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39321): Connection reset by peer (error 104)
2026-09-28T15:56:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2151 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:04 dist-test-slave-1vw0 krb5kdc[12071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:04 dist-test-slave-1vw0 krb5kdc[12071](info): set up 2 sockets
Sep 28 15:56:04 dist-test-slave-1vw0 krb5kdc[12071](info): commencing operation
krb5kdc: starting...
W20260928 15:56:05.297595 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:05 dist-test-slave-1vw0 krb5kdc[12071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:56:05.604605 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:05.953293 12084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:05.953667 12084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:05.953842 12084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:05.961187 12084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:05.961367 12084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:05.961488 12084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:05.961601 12084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:05.961690 12084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:05.971870 12084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:05.973299 12084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:05.974709 12084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:05.987174 12089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:05.987128 12092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:05.987216 12090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:05.988648 12084 server_base.cc:1034] running on GCE node
I20260928 15:56:05.989507 12084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:05.991001 12084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:05.992257 12084 hybrid_clock.cc:648] HybridClock initialized: now 1790610965992215 us; error 56 us; skew 500 ppm
I20260928 15:56:05.992636 12084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:05 dist-test-slave-1vw0 krb5kdc[12071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:05.996301 12084 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:56:05.997888 12084 webserver.cc:460] Webserver started at http://127.0.0.1:36335/ using document root <none> and password file <none>
I20260928 15:56:05.998627 12084 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:05.998756 12084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:05.999045 12084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:06.002327 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "e34b4a8fb94140a1ada156ad100ad734"
format_stamp: "Formatted at 2026-09-28 15:56:05 on dist-test-slave-1vw0"
server_key: "fd522b7dd9eff59eb4ada72c979a1475"
server_key_iv: "f98363bfe5953f2dc2faf39fc346c945"
server_key_version: "encryptionkey@0"
I20260928 15:56:06.003188 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "e34b4a8fb94140a1ada156ad100ad734"
format_stamp: "Formatted at 2026-09-28 15:56:05 on dist-test-slave-1vw0"
server_key: "fd522b7dd9eff59eb4ada72c979a1475"
server_key_iv: "f98363bfe5953f2dc2faf39fc346c945"
server_key_version: "encryptionkey@0"
I20260928 15:56:06.009125 12084 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 15:56:06.012908 12099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:06.014379 12084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:56:06.014616 12084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "e34b4a8fb94140a1ada156ad100ad734"
format_stamp: "Formatted at 2026-09-28 15:56:05 on dist-test-slave-1vw0"
server_key: "fd522b7dd9eff59eb4ada72c979a1475"
server_key_iv: "f98363bfe5953f2dc2faf39fc346c945"
server_key_version: "encryptionkey@0"
I20260928 15:56:06.014838 12084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:06.027923 12084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:06.032531 12084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:06.032862 12084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:06.051997 12084 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40977
I20260928 15:56:06.052021 12150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40977 every 8 connection(s)
I20260928 15:56:06.053557 12084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:06.054039 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 12084
I20260928 15:56:06.054243 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:06.054922 23159 external_mini_cluster.cc:1428] Setting key d7780157f3c5dfb49e878d06bdb03e5f
I20260928 15:56:06.058118 12151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:06.069209 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:06.056868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:34851 (local address 127.0.0.1:40977)
0928 15:56:06.057615 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:06.057627 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:06.057678 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:56:06.059407 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:06.059429 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:06.059819 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:06.060188 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:06.060824 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:06.062142 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:06.063511 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:06.064177 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:06.064381 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:06.065720 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:06.065734 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:06.065751 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:06.066515 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:06.066629 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:06.066684 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:06.067297 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:06.067484 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 15:56:06.067705 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":305,"threads_started":1}
I20260928 15:56:06.070568 12151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734: Bootstrap starting.
I20260928 15:56:06.073740 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:06.074729 12151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:06.076056 12151 log.cc:824] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:06.079144 12151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734: No bootstrap required, opened a new log
I20260928 15:56:06.086450 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:06.079311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:40735 (local address 127.0.0.1:40977)
0928 15:56:06.079516 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:06.079525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:06.080021 (+ 496us) server_negotiation.cc:407] Connection header received
0928 15:56:06.080209 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:06.080212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:06.080274 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:06.080359 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:06.081069 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:06.081886 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:06.083781 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:06.084220 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:06.084317 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:06.085618 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:06.085622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:06.085626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:06.085647 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:06.085689 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:06.085704 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:06.085790 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:06.086031 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 15:56:06.086114 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 15:56:06.089061 12151 raft_consensus.cc:348] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34b4a8fb94140a1ada156ad100ad734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260928 15:56:06.089498 12151 raft_consensus.cc:374] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:06.089609 12151 raft_consensus.cc:729] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e34b4a8fb94140a1ada156ad100ad734, State: Initialized, Role: FOLLOWER
I20260928 15:56:06.090194 12151 consensus_queue.cc:260] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34b4a8fb94140a1ada156ad100ad734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260928 15:56:06.090443 12151 raft_consensus.cc:388] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:06.090595 12151 raft_consensus.cc:482] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:06.090777 12151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:06.093037 12151 raft_consensus.cc:504] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34b4a8fb94140a1ada156ad100ad734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260928 15:56:06.093564 12151 leader_election.cc:302] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e34b4a8fb94140a1ada156ad100ad734; no voters:
I20260928 15:56:06.094244 12151 leader_election.cc:288] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:06.094440 12161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:06.095465 12161 raft_consensus.cc:686] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [term 1 LEADER]: Becoming Leader. State: Replica: e34b4a8fb94140a1ada156ad100ad734, State: Running, Role: LEADER
I20260928 15:56:06.096129 12161 consensus_queue.cc:237] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34b4a8fb94140a1ada156ad100ad734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260928 15:56:06.096949 12151 sys_catalog.cc:564] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:06.099531 12162 sys_catalog.cc:455] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e34b4a8fb94140a1ada156ad100ad734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34b4a8fb94140a1ada156ad100ad734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260928 15:56:06.099932 12163 sys_catalog.cc:455] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e34b4a8fb94140a1ada156ad100ad734. Latest consensus state: current_term: 1 leader_uuid: "e34b4a8fb94140a1ada156ad100ad734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e34b4a8fb94140a1ada156ad100ad734" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260928 15:56:06.100292 12163 sys_catalog.cc:458] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:06.100263 12162 sys_catalog.cc:458] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:06.106375 12169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:06.112262 12169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:06.121831 12169 catalog_manager.cc:1347] Generated new cluster ID: 2ffab0f369b04cf18188d5c6772cd38d
I20260928 15:56:06.121968 12169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:06.136399 12169 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:06.137913 12169 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:06.145707 12169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e34b4a8fb94140a1ada156ad100ad734: Generated new TSK 0
I20260928 15:56:06.146548 12169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:06.154263 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12084
W20260928 15:56:06.155406 12146 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:34851
W20260928 15:56:06.156157 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12084: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:06.156299 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 12084
W20260928 15:56:06.174525 12069 connection.cc:535] client connection to 127.0.0.1:40977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40977): Connection reset by peer (error 104)
2026-09-28T15:56:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:06 dist-test-slave-1vw0 krb5kdc[12182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:06 dist-test-slave-1vw0 krb5kdc[12182](info): set up 2 sockets
Sep 28 15:56:06 dist-test-slave-1vw0 krb5kdc[12182](info): commencing operation
krb5kdc: starting...
W20260928 15:56:07.298233 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.004s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:07 dist-test-slave-1vw0 krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:56:07.604106 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:07.970783 12195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:07.971102 12195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:07.971266 12195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:07.978920 12195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:07.979059 12195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:07.979151 12195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:07.979290 12195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:07.979377 12195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:07.989912 12195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:07.991438 12195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:07.993033 12195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:08.006472 12201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:08.006472 12200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:08.007119 12203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:08.007514 12195 server_base.cc:1034] running on GCE node
I20260928 15:56:08.008391 12195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:08.010259 12195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:08.011521 12195 hybrid_clock.cc:648] HybridClock initialized: now 1790610968011479 us; error 65 us; skew 500 ppm
I20260928 15:56:08.011927 12195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:08 dist-test-slave-1vw0 krb5kdc[12182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:08.015479 12195 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:56:08.017282 12195 webserver.cc:460] Webserver started at http://127.0.0.1:41159/ using document root <none> and password file <none>
I20260928 15:56:08.018009 12195 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:08.018172 12195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:08.018499 12195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:08.021854 12195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "9ae9809b082e42fda8395d576e09eb2d"
format_stamp: "Formatted at 2026-09-28 15:56:08 on dist-test-slave-1vw0"
server_key: "15dac13e7bb3de1c10d4bcd175db0e95"
server_key_iv: "77ecb390c12da81a93110b63e590176b"
server_key_version: "encryptionkey@0"
I20260928 15:56:08.022720 12195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "9ae9809b082e42fda8395d576e09eb2d"
format_stamp: "Formatted at 2026-09-28 15:56:08 on dist-test-slave-1vw0"
server_key: "15dac13e7bb3de1c10d4bcd175db0e95"
server_key_iv: "77ecb390c12da81a93110b63e590176b"
server_key_version: "encryptionkey@0"
I20260928 15:56:08.028492 12195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 15:56:08.032598 12210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:08.034390 12195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:56:08.034626 12195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "9ae9809b082e42fda8395d576e09eb2d"
format_stamp: "Formatted at 2026-09-28 15:56:08 on dist-test-slave-1vw0"
server_key: "15dac13e7bb3de1c10d4bcd175db0e95"
server_key_iv: "77ecb390c12da81a93110b63e590176b"
server_key_version: "encryptionkey@0"
I20260928 15:56:08.034868 12195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:08.041473 12195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:08.046424 12195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:08.046777 12195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:08.066991 12195 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43867
I20260928 15:56:08.067020 12261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43867 every 8 connection(s)
I20260928 15:56:08.068585 12195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:08.072664 12262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:08.074779 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 12195
I20260928 15:56:08.075009 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:08.075573 23159 external_mini_cluster.cc:1428] Setting key 3ff0eb145199f4363afe96fb5ff124bf
I20260928 15:56:08.084313 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:08.077560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:37751 (local address 127.0.0.1:43867)
0928 15:56:08.078474 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:08.078486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:08.078535 (+ 49us) server_negotiation.cc:407] Connection header received
0928 15:56:08.080348 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:08.080377 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:08.080865 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:08.081257 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:08.082434 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:08.082441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:08.082504 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:08.082527 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:08.082618 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:08.082721 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:08.082951 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:08.083058 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 15:56:08.083310 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":408,"threads_started":1}
I20260928 15:56:08.088207 12262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d: Bootstrap starting.
I20260928 15:56:08.088430 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:08.091683 12262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:08.093091 12262 log.cc:824] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:08.095741 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:08.093042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:55147 (local address 127.0.0.1:43867)
0928 15:56:08.093252 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:08.093258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:08.093585 (+ 327us) server_negotiation.cc:407] Connection header received
0928 15:56:08.093830 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:08.093834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:08.093892 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:08.094012 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:08.095008 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:08.095013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:08.095018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:08.095037 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:08.095085 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:08.095101 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:08.095229 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:08.095367 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 15:56:08.095436 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 15:56:08.096309 12262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d: No bootstrap required, opened a new log
I20260928 15:56:08.104199 12262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9809b082e42fda8395d576e09eb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260928 15:56:08.104691 12262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:08.104851 12262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae9809b082e42fda8395d576e09eb2d, State: Initialized, Role: FOLLOWER
I20260928 15:56:08.105505 12262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9809b082e42fda8395d576e09eb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260928 15:56:08.105783 12262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:08.105950 12262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:08.106138 12262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:08.108502 12262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9809b082e42fda8395d576e09eb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260928 15:56:08.109145 12262 leader_election.cc:302] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ae9809b082e42fda8395d576e09eb2d; no voters:
I20260928 15:56:08.109855 12262 leader_election.cc:288] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:08.110001 12272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:08.110755 12272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [term 1 LEADER]: Becoming Leader. State: Replica: 9ae9809b082e42fda8395d576e09eb2d, State: Running, Role: LEADER
I20260928 15:56:08.111338 12272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9809b082e42fda8395d576e09eb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260928 15:56:08.112190 12262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:08.114645 12273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ae9809b082e42fda8395d576e09eb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9809b082e42fda8395d576e09eb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } } }
I20260928 15:56:08.114642 12274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae9809b082e42fda8395d576e09eb2d. Latest consensus state: current_term: 1 leader_uuid: "9ae9809b082e42fda8395d576e09eb2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae9809b082e42fda8395d576e09eb2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } } }
I20260928 15:56:08.115124 12273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:08.115171 12274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:08.117084 12279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:08.123534 12279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:08.134143 12279 catalog_manager.cc:1347] Generated new cluster ID: ad2e35e9840148038f07a8174d369c0c
I20260928 15:56:08.134307 12279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:08.158245 12279 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:08.158820 12279 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:08.176827 12279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ae9809b082e42fda8395d576e09eb2d: Generated new TSK 0
I20260928 15:56:08.177825 12279 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:08.192544 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12195
W20260928 15:56:08.193823 12257 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:37751
W20260928 15:56:08.194797 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12195: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:08.195037 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 12195
2026-09-28T15:56:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2040 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:08 dist-test-slave-1vw0 krb5kdc[12293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:08 dist-test-slave-1vw0 krb5kdc[12293](info): set up 2 sockets
Sep 28 15:56:08 dist-test-slave-1vw0 krb5kdc[12293](info): commencing operation
krb5kdc: starting...
W20260928 15:56:09.341881 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:09 dist-test-slave-1vw0 krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:56:09.652592 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:10.002089 12306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:10.002398 12306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:10.002595 12306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:10.009963 12306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:10.010090 12306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:10.010214 12306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:10.010336 12306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:10.010416 12306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:10.020802 12306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:10.022214 12306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:10.023615 12306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:10.036548 12311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:10.036710 12312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:10.036949 12314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:10.038136 12306 server_base.cc:1034] running on GCE node
I20260928 15:56:10.038993 12306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:10.040495 12306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:10.041707 12306 hybrid_clock.cc:648] HybridClock initialized: now 1790610970041644 us; error 56 us; skew 500 ppm
I20260928 15:56:10.042037 12306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:10 dist-test-slave-1vw0 krb5kdc[12293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:10.053038 12306 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:56:10.054687 12306 webserver.cc:460] Webserver started at http://127.0.0.1:43155/ using document root <none> and password file <none>
I20260928 15:56:10.055387 12306 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:10.055496 12306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:10.055769 12306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:10.058915 12306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "ce2b94d9687346e1a1047e4db40c34aa"
format_stamp: "Formatted at 2026-09-28 15:56:10 on dist-test-slave-1vw0"
server_key: "dc3139c05fcef4a7c09b44ba48f5e9e9"
server_key_iv: "20faf4f1d7500269d7427d4efe831024"
server_key_version: "encryptionkey@0"
I20260928 15:56:10.059689 12306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "ce2b94d9687346e1a1047e4db40c34aa"
format_stamp: "Formatted at 2026-09-28 15:56:10 on dist-test-slave-1vw0"
server_key: "dc3139c05fcef4a7c09b44ba48f5e9e9"
server_key_iv: "20faf4f1d7500269d7427d4efe831024"
server_key_version: "encryptionkey@0"
I20260928 15:56:10.065205 12306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 15:56:10.069365 12321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:10.070828 12306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:56:10.070981 12306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "ce2b94d9687346e1a1047e4db40c34aa"
format_stamp: "Formatted at 2026-09-28 15:56:10 on dist-test-slave-1vw0"
server_key: "dc3139c05fcef4a7c09b44ba48f5e9e9"
server_key_iv: "20faf4f1d7500269d7427d4efe831024"
server_key_version: "encryptionkey@0"
I20260928 15:56:10.071224 12306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:10.102464 12306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:10.107048 12306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:10.107295 12306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:10.127708 12306 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42963
I20260928 15:56:10.127728 12372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42963 every 8 connection(s)
I20260928 15:56:10.129383 12306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:10.132712 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 12306
I20260928 15:56:10.132958 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:10.133455 23159 external_mini_cluster.cc:1428] Setting key f61b13ea75e4de8deab16e9062dfc3c3
I20260928 15:56:10.133821 12373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:10.149600 12373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa: Bootstrap starting.
Sep 28 15:56:10 dist-test-slave-1vw0 krb5kdc[12293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 15:56:10.153781 12373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:10.155356 12373 log.cc:824] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:10.159567 12373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa: No bootstrap required, opened a new log
I20260928 15:56:10.161821 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:10.135657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:55965 (local address 127.0.0.1:42963)
0928 15:56:10.136766 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:10.136784 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:10.137226 (+ 442us) server_negotiation.cc:407] Connection header received
0928 15:56:10.139353 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:10.139393 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:10.140082 (+ 689us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:10.140647 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:10.142228 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:10.144191 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:10.145673 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:10.146879 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:10.147101 (+ 222us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:10.151584 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:10.151597 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:10.151619 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:10.151671 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:10.156073 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:10.157519 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:10.157528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:10.157537 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:10.157759 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:10.158324 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:10.158334 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:10.158338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:10.159126 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:10.159460 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:10.159948 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 15:56:10.160364 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":441,"threads_started":1}
I20260928 15:56:10.166145 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:10.168536 12373 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2b94d9687346e1a1047e4db40c34aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260928 15:56:10.169126 12373 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:10.169258 12373 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2b94d9687346e1a1047e4db40c34aa, State: Initialized, Role: FOLLOWER
I20260928 15:56:10.169986 12373 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2b94d9687346e1a1047e4db40c34aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260928 15:56:10.170240 12373 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:10.170389 12373 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:10.170612 12373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:10.173472 12373 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2b94d9687346e1a1047e4db40c34aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260928 15:56:10.174217 12373 leader_election.cc:302] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce2b94d9687346e1a1047e4db40c34aa; no voters:
I20260928 15:56:10.175011 12373 leader_election.cc:288] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:10.175128 12383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:10.177366 12373 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:10.177377 12383 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [term 1 LEADER]: Becoming Leader. State: Replica: ce2b94d9687346e1a1047e4db40c34aa, State: Running, Role: LEADER
I20260928 15:56:10.178198 12383 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2b94d9687346e1a1047e4db40c34aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260928 15:56:10.181527 12384 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce2b94d9687346e1a1047e4db40c34aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2b94d9687346e1a1047e4db40c34aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } } }
I20260928 15:56:10.182111 12384 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:10.181932 12385 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce2b94d9687346e1a1047e4db40c34aa. Latest consensus state: current_term: 1 leader_uuid: "ce2b94d9687346e1a1047e4db40c34aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce2b94d9687346e1a1047e4db40c34aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } } }
I20260928 15:56:10.182276 12385 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:10.188861 12390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:10.191365 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:10.170780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:36953 (local address 127.0.0.1:42963)
0928 15:56:10.170998 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:10.171007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:10.171576 (+ 569us) server_negotiation.cc:407] Connection header received
0928 15:56:10.171703 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:10.171709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:10.171806 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:10.171925 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:10.173502 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:10.174595 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:10.177073 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:10.177847 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:10.177995 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:10.182392 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:10.182401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:10.182411 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:10.182471 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:10.187485 (+ 5014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:10.188974 (+ 1489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:10.188982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:10.188988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:10.189193 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:10.189724 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:10.189727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:10.189729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:10.190071 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:10.190302 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:10.190748 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 15:56:10.190938 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:56:10.194696 12390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:10.205114 12390 catalog_manager.cc:1347] Generated new cluster ID: 7b1ba59217f045cb9fbfd15f737f22f6
I20260928 15:56:10.205259 12390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:10.248674 12390 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:10.250756 12390 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:10.264750 12390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce2b94d9687346e1a1047e4db40c34aa: Generated new TSK 0
I20260928 15:56:10.265877 12390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:10.277356 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12306
I20260928 15:56:10.279171 12368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:10.377022 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 12306
W20260928 15:56:10.398723 12291 connection.cc:535] client connection to 127.0.0.1:42963 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42963): Connection reset by peer (error 104)
2026-09-28T15:56:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2183 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:10 dist-test-slave-1vw0 krb5kdc[12405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:10 dist-test-slave-1vw0 krb5kdc[12405](info): set up 2 sockets
Sep 28 15:56:10 dist-test-slave-1vw0 krb5kdc[12405](info): commencing operation
krb5kdc: starting...
W20260928 15:56:11.541697 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:11 dist-test-slave-1vw0 krb5kdc[12405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:56:11.858349 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:12.206466 12418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:12.206789 12418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:12.206969 12418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:12.215103 12418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:12.215246 12418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:12.215355 12418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:12.215447 12418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:12.215512 12418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:12.226024 12418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:12.227442 12418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:12.228832 12418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:12.241312 12424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:12.241345 12426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:12.241588 12423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:12.242374 12418 server_base.cc:1034] running on GCE node
I20260928 15:56:12.243186 12418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:12.244699 12418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:12.246006 12418 hybrid_clock.cc:648] HybridClock initialized: now 1790610972245948 us; error 64 us; skew 500 ppm
I20260928 15:56:12.246385 12418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:12 dist-test-slave-1vw0 krb5kdc[12405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:12.249573 12418 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:56:12.251075 12418 webserver.cc:460] Webserver started at http://127.0.0.1:37101/ using document root <none> and password file <none>
I20260928 15:56:12.251740 12418 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:12.251842 12418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:12.252084 12418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:12.255147 12418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "9f79cfe4c72d455cab72a245983e6366"
format_stamp: "Formatted at 2026-09-28 15:56:12 on dist-test-slave-1vw0"
server_key: "9d5d2dda2d498876cea4320583e9a65f"
server_key_iv: "1d7090956456107ca37743a2cae2ad6a"
server_key_version: "encryptionkey@0"
I20260928 15:56:12.255887 12418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "9f79cfe4c72d455cab72a245983e6366"
format_stamp: "Formatted at 2026-09-28 15:56:12 on dist-test-slave-1vw0"
server_key: "9d5d2dda2d498876cea4320583e9a65f"
server_key_iv: "1d7090956456107ca37743a2cae2ad6a"
server_key_version: "encryptionkey@0"
I20260928 15:56:12.261431 12418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:56:12.265131 12433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:12.266594 12418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:56:12.266750 12418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "9f79cfe4c72d455cab72a245983e6366"
format_stamp: "Formatted at 2026-09-28 15:56:12 on dist-test-slave-1vw0"
server_key: "9d5d2dda2d498876cea4320583e9a65f"
server_key_iv: "1d7090956456107ca37743a2cae2ad6a"
server_key_version: "encryptionkey@0"
I20260928 15:56:12.266929 12418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:12.303906 12418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:12.308521 12418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:12.308796 12418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:12.328536 12418 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37305
I20260928 15:56:12.328572 12484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37305 every 8 connection(s)
I20260928 15:56:12.330024 12418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:12.333863 12485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:12.338851 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 12418
I20260928 15:56:12.339046 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:12.339478 23159 external_mini_cluster.cc:1428] Setting key b77707f00763a25ce48e182fa9c38c75
I20260928 15:56:12.346726 12485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366: Bootstrap starting.
I20260928 15:56:12.350591 12485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:12.351981 12485 log.cc:824] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:12.355610 12485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366: No bootstrap required, opened a new log
I20260928 15:56:12.355631 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:12.341481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:46947 (local address 127.0.0.1:37305)
0928 15:56:12.342363 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:12.342380 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:12.342442 (+ 62us) server_negotiation.cc:407] Connection header received
0928 15:56:12.345108 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:12.345140 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:12.345749 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:12.346237 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:12.346803 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:12.348899 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:12.350261 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:12.351386 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:12.351569 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:12.352833 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:12.352842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:12.352859 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:12.353526 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:12.353632 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:12.353664 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:12.353818 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:12.354102 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 15:56:12.354319 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":353,"threads_started":1}
I20260928 15:56:12.359619 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:12.364817 12485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f79cfe4c72d455cab72a245983e6366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260928 15:56:12.365433 12485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:12.365581 12485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f79cfe4c72d455cab72a245983e6366, State: Initialized, Role: FOLLOWER
I20260928 15:56:12.366314 12485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f79cfe4c72d455cab72a245983e6366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260928 15:56:12.366605 12485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:12.366773 12485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:12.366953 12485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:12.369853 12485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f79cfe4c72d455cab72a245983e6366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260928 15:56:12.370479 12485 leader_election.cc:302] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f79cfe4c72d455cab72a245983e6366; no voters:
I20260928 15:56:12.371182 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:12.364029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:54859 (local address 127.0.0.1:37305)
0928 15:56:12.364198 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:12.364203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:12.364713 (+ 510us) server_negotiation.cc:407] Connection header received
0928 15:56:12.364946 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:12.364951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:12.365036 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:12.365141 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:12.365754 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:12.366887 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:12.368223 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:12.368975 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:12.369105 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:12.370265 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:12.370270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:12.370274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:12.370294 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:12.370367 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:12.370391 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:12.370532 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:12.370711 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 15:56:12.370763 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:56:12.371275 12485 leader_election.cc:288] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:12.371469 12495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:12.372140 12495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [term 1 LEADER]: Becoming Leader. State: Replica: 9f79cfe4c72d455cab72a245983e6366, State: Running, Role: LEADER
I20260928 15:56:12.372826 12495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f79cfe4c72d455cab72a245983e6366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260928 15:56:12.374476 12485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:12.375788 12496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f79cfe4c72d455cab72a245983e6366. Latest consensus state: current_term: 1 leader_uuid: "9f79cfe4c72d455cab72a245983e6366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f79cfe4c72d455cab72a245983e6366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260928 15:56:12.375828 12497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f79cfe4c72d455cab72a245983e6366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f79cfe4c72d455cab72a245983e6366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260928 15:56:12.376185 12496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:12.376212 12497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:12.381554 12500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:12.388849 12500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:12.399323 12500 catalog_manager.cc:1347] Generated new cluster ID: ef4dca6b46184fb8a3c539ece849f3f1
I20260928 15:56:12.399461 12500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:12.418447 12500 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:12.419840 12500 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:12.428064 12500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f79cfe4c72d455cab72a245983e6366: Generated new TSK 0
I20260928 15:56:12.429093 12500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:12.442338 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12418
W20260928 15:56:12.443682 12480 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:46947
W20260928 15:56:12.444563 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12418: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:12.444785 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 12418
W20260928 15:56:12.464409 12403 connection.cc:535] client connection to 127.0.0.1:37305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37305): Connection reset by peer (error 104)
2026-09-28T15:56:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2065 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:12 dist-test-slave-1vw0 krb5kdc[12516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:12 dist-test-slave-1vw0 krb5kdc[12516](info): set up 2 sockets
Sep 28 15:56:12 dist-test-slave-1vw0 krb5kdc[12516](info): commencing operation
krb5kdc: starting...
W20260928 15:56:13.581683 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:13 dist-test-slave-1vw0 krb5kdc[12516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 15:56:13.895960 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:14.248829 12529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:14.249215 12529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:14.249388 12529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:14.256978 12529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:14.257160 12529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:14.257277 12529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:14.257401 12529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:14.257500 12529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:14.267761 12529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:14.269236 12529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:14.270646 12529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:14.283313 12537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:14.283299 12534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:14.283288 12535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:14.284111 12529 server_base.cc:1034] running on GCE node
I20260928 15:56:14.285252 12529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:14.286870 12529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:14.288131 12529 hybrid_clock.cc:648] HybridClock initialized: now 1790610974288082 us; error 74 us; skew 500 ppm
I20260928 15:56:14.288530 12529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:14 dist-test-slave-1vw0 krb5kdc[12516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:14.292196 12529 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 15:56:14.293915 12529 webserver.cc:460] Webserver started at http://127.0.0.1:44563/ using document root <none> and password file <none>
I20260928 15:56:14.294736 12529 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:14.294914 12529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:14.295351 12529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:14.298827 12529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "0c43044c978e422bbf7e05393ce6e027"
format_stamp: "Formatted at 2026-09-28 15:56:14 on dist-test-slave-1vw0"
server_key: "02bfa934e7f205b0b0b500c64b68b8c0"
server_key_iv: "f138c30b1d61a39392700f47333076e6"
server_key_version: "encryptionkey@0"
I20260928 15:56:14.299713 12529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "0c43044c978e422bbf7e05393ce6e027"
format_stamp: "Formatted at 2026-09-28 15:56:14 on dist-test-slave-1vw0"
server_key: "02bfa934e7f205b0b0b500c64b68b8c0"
server_key_iv: "f138c30b1d61a39392700f47333076e6"
server_key_version: "encryptionkey@0"
I20260928 15:56:14.305706 12529 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 15:56:14.309473 12544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:14.310959 12529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:56:14.311184 12529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "0c43044c978e422bbf7e05393ce6e027"
format_stamp: "Formatted at 2026-09-28 15:56:14 on dist-test-slave-1vw0"
server_key: "02bfa934e7f205b0b0b500c64b68b8c0"
server_key_iv: "f138c30b1d61a39392700f47333076e6"
server_key_version: "encryptionkey@0"
I20260928 15:56:14.311424 12529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:14.317981 12529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:14.322463 12529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:14.322813 12529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:14.342377 12529 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46265
I20260928 15:56:14.342414 12595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46265 every 8 connection(s)
I20260928 15:56:14.343950 12529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:14.346563 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 12529
I20260928 15:56:14.346747 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:14.347285 23159 external_mini_cluster.cc:1428] Setting key 2895831ecdd82f9a9a9f2aec614292ea
I20260928 15:56:14.348438 12596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:14.355319 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:14.349182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:51429 (local address 127.0.0.1:46265)
0928 15:56:14.349974 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:14.349993 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:14.350034 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:56:14.351583 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:14.351606 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:14.351973 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:14.352329 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:14.353549 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:14.353556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:14.353568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:14.353593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:14.353673 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:14.353698 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:14.353876 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:14.354063 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 15:56:14.354368 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":367,"threads_started":1}
I20260928 15:56:14.359287 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:14.362207 12596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027: Bootstrap starting.
I20260928 15:56:14.366382 12596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:14.366474 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:14.363881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.235:35327 (local address 127.0.0.1:46265)
0928 15:56:14.364113 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:14.364119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:14.364431 (+ 312us) server_negotiation.cc:407] Connection header received
0928 15:56:14.364632 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:14.364636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:14.364687 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:14.364844 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:14.365763 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:14.365768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:14.365771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:14.365785 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:14.365825 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:14.365840 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:14.365925 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:14.366075 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 15:56:14.366132 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 15:56:14.367774 12596 log.cc:824] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:14.370725 12596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027: No bootstrap required, opened a new log
I20260928 15:56:14.378110 12596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43044c978e422bbf7e05393ce6e027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260928 15:56:14.378515 12596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:14.378590 12596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c43044c978e422bbf7e05393ce6e027, State: Initialized, Role: FOLLOWER
I20260928 15:56:14.379169 12596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43044c978e422bbf7e05393ce6e027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260928 15:56:14.379365 12596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:14.379467 12596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:14.379592 12596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:14.381819 12596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43044c978e422bbf7e05393ce6e027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260928 15:56:14.382400 12596 leader_election.cc:302] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c43044c978e422bbf7e05393ce6e027; no voters:
I20260928 15:56:14.383008 12596 leader_election.cc:288] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:14.383180 12606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:14.383889 12606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [term 1 LEADER]: Becoming Leader. State: Replica: 0c43044c978e422bbf7e05393ce6e027, State: Running, Role: LEADER
I20260928 15:56:14.384403 12606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43044c978e422bbf7e05393ce6e027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260928 15:56:14.385298 12596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:14.388922 12607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c43044c978e422bbf7e05393ce6e027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43044c978e422bbf7e05393ce6e027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } } }
I20260928 15:56:14.389447 12607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:14.390023 12608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c43044c978e422bbf7e05393ce6e027. Latest consensus state: current_term: 1 leader_uuid: "0c43044c978e422bbf7e05393ce6e027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c43044c978e422bbf7e05393ce6e027" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } } }
I20260928 15:56:14.390372 12608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:14.390405 12612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:14.402024 12612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:14.419770 12612 catalog_manager.cc:1347] Generated new cluster ID: 7795c574e33f4694bc05036374448a70
I20260928 15:56:14.419905 12612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:14.435518 12612 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:14.436031 12612 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:14.460786 12612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c43044c978e422bbf7e05393ce6e027: Generated new TSK 0
I20260928 15:56:14.462069 12612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:14.473389 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12529
W20260928 15:56:14.474781 12591 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.235:51429
W20260928 15:56:14.475582 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12529: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:14.475757 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 12529
2026-09-28T15:56:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:14 dist-test-slave-1vw0 krb5kdc[12627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:14 dist-test-slave-1vw0 krb5kdc[12627](info): set up 2 sockets
Sep 28 15:56:14 dist-test-slave-1vw0 krb5kdc[12627](info): commencing operation
krb5kdc: starting...
W20260928 15:56:15.613790 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:15 dist-test-slave-1vw0 krb5kdc[12627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:56:15.926548 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43895
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:43895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:16.273037 12640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:16.273350 12640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:16.273527 12640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:16.281126 12640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:16.281261 12640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:16.281359 12640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:16.281479 12640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:16.281567 12640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:16.291956 12640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:43895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:16.293483 12640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:16.294929 12640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:16.308144 12645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:16.308050 12646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:16.308097 12648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:16.309091 12640 server_base.cc:1034] running on GCE node
I20260928 15:56:16.310070 12640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:16.311764 12640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:16.313122 12640 hybrid_clock.cc:648] HybridClock initialized: now 1790610976313021 us; error 61 us; skew 500 ppm
I20260928 15:56:16.313571 12640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:16 dist-test-slave-1vw0 krb5kdc[12627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610976, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:16.324510 12640 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:56:16.326447 12640 webserver.cc:460] Webserver started at http://127.22.157.254:38125/ using document root <none> and password file <none>
I20260928 15:56:16.327212 12640 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:16.327327 12640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:16.327606 12640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:16.331009 12640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "1e728639a9074384b14aeb223d8875a2"
format_stamp: "Formatted at 2026-09-28 15:56:16 on dist-test-slave-1vw0"
server_key: "48bf7ee999b7a17dedbf7ff761d46363"
server_key_iv: "dd0293136a30168756039acc12a668de"
server_key_version: "encryptionkey@0"
I20260928 15:56:16.331986 12640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "1e728639a9074384b14aeb223d8875a2"
format_stamp: "Formatted at 2026-09-28 15:56:16 on dist-test-slave-1vw0"
server_key: "48bf7ee999b7a17dedbf7ff761d46363"
server_key_iv: "dd0293136a30168756039acc12a668de"
server_key_version: "encryptionkey@0"
I20260928 15:56:16.338014 12640 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 15:56:16.341894 12655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:16.343515 12640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260928 15:56:16.343779 12640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "1e728639a9074384b14aeb223d8875a2"
format_stamp: "Formatted at 2026-09-28 15:56:16 on dist-test-slave-1vw0"
server_key: "48bf7ee999b7a17dedbf7ff761d46363"
server_key_iv: "dd0293136a30168756039acc12a668de"
server_key_version: "encryptionkey@0"
I20260928 15:56:16.343997 12640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:16.364146 12640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:16.368630 12640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:16.368988 12640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:16.389948 12640 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:43895
I20260928 15:56:16.389971 12706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:43895 every 8 connection(s)
I20260928 15:56:16.391515 12640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:16.395622 12707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:16.396829 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 12640
I20260928 15:56:16.397020 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:16.397581 23159 external_mini_cluster.cc:1428] Setting key 629554c3b39d8b57c79555dd4bfe4949
I20260928 15:56:16.411096 12707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2: Bootstrap starting.
Sep 28 15:56:16 dist-test-slave-1vw0 krb5kdc[12627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:56:16.415162 12707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:16.416553 12707 log.cc:824] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:16.419857 12707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2: No bootstrap required, opened a new log
I20260928 15:56:16.423221 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:16.399530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37849 (local address 127.22.157.254:43895)
0928 15:56:16.400426 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:16.400438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:16.400486 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:56:16.402415 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:16.402444 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:16.402922 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:16.403289 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:16.405325 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:16.406729 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:16.408682 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:16.409570 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:16.409777 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:16.414894 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:16.414913 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:16.414930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:16.414975 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:16.418439 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:16.419400 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:16.419409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:16.419416 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:16.419597 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:16.420176 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:16.420180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:16.420182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:16.420809 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:16.421174 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:16.421708 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 15:56:16.422090 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":400,"threads_started":1}
I20260928 15:56:16.427812 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:16.429062 12707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e728639a9074384b14aeb223d8875a2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43895 } }
I20260928 15:56:16.429499 12707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:16.429607 12707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e728639a9074384b14aeb223d8875a2, State: Initialized, Role: FOLLOWER
I20260928 15:56:16.430238 12707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e728639a9074384b14aeb223d8875a2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43895 } }
I20260928 15:56:16.430490 12707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:16.430709 12707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:16.430904 12707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:16.435000 12707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e728639a9074384b14aeb223d8875a2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43895 } }
I20260928 15:56:16.435797 12707 leader_election.cc:302] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e728639a9074384b14aeb223d8875a2; no voters:
I20260928 15:56:16.436506 12707 leader_election.cc:288] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:16.436628 12717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:16.438252 12717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1e728639a9074384b14aeb223d8875a2, State: Running, Role: LEADER
I20260928 15:56:16.439158 12707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:16.439023 12717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e728639a9074384b14aeb223d8875a2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43895 } }
I20260928 15:56:16.442106 12719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e728639a9074384b14aeb223d8875a2. Latest consensus state: current_term: 1 leader_uuid: "1e728639a9074384b14aeb223d8875a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e728639a9074384b14aeb223d8875a2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43895 } } }
I20260928 15:56:16.442557 12719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:16.443238 12718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e728639a9074384b14aeb223d8875a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e728639a9074384b14aeb223d8875a2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43895 } } }
I20260928 15:56:16.443568 12718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:16.445689 12723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:16.451473 12723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:16.458108 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:16.432875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33503 (local address 127.22.157.254:43895)
0928 15:56:16.433071 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:16.433076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:16.435927 (+ 2851us) server_negotiation.cc:407] Connection header received
0928 15:56:16.436215 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:16.436222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:16.436330 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:16.436451 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:16.438548 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:16.439823 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:16.442261 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:16.443300 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:16.443476 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:16.448086 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:16.448097 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:16.448105 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:16.448165 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:16.453585 (+ 5420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:16.455185 (+ 1600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:16.455194 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:16.455203 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:16.455416 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:16.456173 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:16.456179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:16.456183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:16.456640 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:16.456959 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:16.457477 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 15:56:16.457693 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:56:16.463693 12723 catalog_manager.cc:1347] Generated new cluster ID: 98d7553442b548258bfb63e0d9c959f5
I20260928 15:56:16.463894 12723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:16.501475 12723 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:16.503443 12723 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:16.512001 12723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e728639a9074384b14aeb223d8875a2: Generated new TSK 0
I20260928 15:56:16.513273 12723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:16.521548 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12640
I20260928 15:56:16.522786 12702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:16.619594 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 12640
W20260928 15:56:16.639431 12625 connection.cc:535] client connection to 127.22.157.254:43895 recv error: Network error: failed to read from TLS socket (remote: 127.22.157.254:43895): Connection reset by peer (error 104)
2026-09-28T15:56:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2148 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:16 dist-test-slave-1vw0 krb5kdc[12740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:16 dist-test-slave-1vw0 krb5kdc[12740](info): set up 2 sockets
Sep 28 15:56:16 dist-test-slave-1vw0 krb5kdc[12740](info): commencing operation
krb5kdc: starting...
W20260928 15:56:17.757630 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:17 dist-test-slave-1vw0 krb5kdc[12740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:56:18.078006 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:36751
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:36751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:18.430506 12753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:18.430863 12753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:18.431052 12753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:18.439687 12753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:18.439826 12753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:18.439925 12753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:18.440021 12753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:18.440085 12753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:18.450897 12753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:36751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:18.452353 12753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:18.453797 12753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:18.466931 12759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:18.466879 12761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:18.466899 12758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:18.468400 12753 server_base.cc:1034] running on GCE node
I20260928 15:56:18.469444 12753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:18.471088 12753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:18.472368 12753 hybrid_clock.cc:648] HybridClock initialized: now 1790610978472319 us; error 54 us; skew 500 ppm
I20260928 15:56:18.472954 12753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:18 dist-test-slave-1vw0 krb5kdc[12740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610978, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:18.476714 12753 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:56:18.478502 12753 webserver.cc:460] Webserver started at http://127.22.157.254:34247/ using document root <none> and password file <none>
I20260928 15:56:18.479336 12753 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:18.479511 12753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:18.479863 12753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:18.483268 12753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "e97e8133e1b7425c8f410f1b568cce25"
format_stamp: "Formatted at 2026-09-28 15:56:18 on dist-test-slave-1vw0"
server_key: "22d0fbeb2a31c74662f80a6bd2445918"
server_key_iv: "70ec8035b4d65bd1b96c095ce95c72fa"
server_key_version: "encryptionkey@0"
I20260928 15:56:18.484144 12753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "e97e8133e1b7425c8f410f1b568cce25"
format_stamp: "Formatted at 2026-09-28 15:56:18 on dist-test-slave-1vw0"
server_key: "22d0fbeb2a31c74662f80a6bd2445918"
server_key_iv: "70ec8035b4d65bd1b96c095ce95c72fa"
server_key_version: "encryptionkey@0"
I20260928 15:56:18.489912 12753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 15:56:18.494027 12768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:18.495801 12753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 15:56:18.496071 12753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "e97e8133e1b7425c8f410f1b568cce25"
format_stamp: "Formatted at 2026-09-28 15:56:18 on dist-test-slave-1vw0"
server_key: "22d0fbeb2a31c74662f80a6bd2445918"
server_key_iv: "70ec8035b4d65bd1b96c095ce95c72fa"
server_key_version: "encryptionkey@0"
I20260928 15:56:18.496351 12753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:18.518657 12753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:18.523576 12753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:18.523891 12753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:18.546114 12753 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:36751
I20260928 15:56:18.546154 12819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:36751 every 8 connection(s)
I20260928 15:56:18.547698 12753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:18.548568 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 12753
I20260928 15:56:18.548856 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:18.549451 23159 external_mini_cluster.cc:1428] Setting key 08fad1c1001bed6c48d22041f86e7332
I20260928 15:56:18.552688 12820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:18.564306 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:18.551486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53023 (local address 127.22.157.254:36751)
0928 15:56:18.552314 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:18.552327 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:18.552372 (+ 45us) server_negotiation.cc:407] Connection header received
0928 15:56:18.554279 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:18.554304 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:18.554712 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:18.555048 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:18.555756 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:18.557237 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:18.559307 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:18.560106 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:18.560261 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:18.561673 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:18.561684 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:18.561699 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:18.562323 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:18.562408 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:18.562440 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:18.562567 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:18.562914 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 15:56:18.563120 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":376,"threads_started":1}
I20260928 15:56:18.566262 12820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25: Bootstrap starting.
I20260928 15:56:18.568770 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:18.570062 12820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:18.571187 12820 log.cc:824] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:18.574856 12820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25: No bootstrap required, opened a new log
I20260928 15:56:18.581176 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:18.573503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50333 (local address 127.22.157.254:36751)
0928 15:56:18.573735 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:18.573745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:18.574349 (+ 604us) server_negotiation.cc:407] Connection header received
0928 15:56:18.574575 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:18.574579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:18.574644 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:18.574730 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:18.575405 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:18.576242 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:18.578332 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:18.578834 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:18.578939 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:18.580250 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:18.580258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:18.580261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:18.580280 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:18.580326 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:18.580342 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:18.580426 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:18.580697 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 15:56:18.580836 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:56:18.584465 12820 raft_consensus.cc:348] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e8133e1b7425c8f410f1b568cce25" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36751 } }
I20260928 15:56:18.585021 12820 raft_consensus.cc:374] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:18.585141 12820 raft_consensus.cc:729] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97e8133e1b7425c8f410f1b568cce25, State: Initialized, Role: FOLLOWER
I20260928 15:56:18.585815 12820 consensus_queue.cc:260] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e8133e1b7425c8f410f1b568cce25" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36751 } }
I20260928 15:56:18.586035 12820 raft_consensus.cc:388] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:18.586144 12820 raft_consensus.cc:482] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:18.586268 12820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:18.588656 12820 raft_consensus.cc:504] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e8133e1b7425c8f410f1b568cce25" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36751 } }
I20260928 15:56:18.589342 12820 leader_election.cc:302] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e97e8133e1b7425c8f410f1b568cce25; no voters:
I20260928 15:56:18.590080 12820 leader_election.cc:288] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:18.590181 12830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:18.590993 12830 raft_consensus.cc:686] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [term 1 LEADER]: Becoming Leader. State: Replica: e97e8133e1b7425c8f410f1b568cce25, State: Running, Role: LEADER
I20260928 15:56:18.591584 12830 consensus_queue.cc:237] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e8133e1b7425c8f410f1b568cce25" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36751 } }
I20260928 15:56:18.592365 12820 sys_catalog.cc:564] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:18.594483 12831 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e97e8133e1b7425c8f410f1b568cce25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e8133e1b7425c8f410f1b568cce25" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36751 } } }
I20260928 15:56:18.596050 12831 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:18.596457 12832 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e97e8133e1b7425c8f410f1b568cce25. Latest consensus state: current_term: 1 leader_uuid: "e97e8133e1b7425c8f410f1b568cce25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e8133e1b7425c8f410f1b568cce25" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 36751 } } }
I20260928 15:56:18.597810 12832 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:18.605412 12837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:18.611631 12837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:18.622682 12837 catalog_manager.cc:1347] Generated new cluster ID: 63eb1b82aeca4cea90e28b067766d0a8
I20260928 15:56:18.622821 12837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:18.635483 12837 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:18.637182 12837 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:18.648216 12837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e97e8133e1b7425c8f410f1b568cce25: Generated new TSK 0
I20260928 15:56:18.649353 12837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:18.666744 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12753
W20260928 15:56:18.668409 12815 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53023
W20260928 15:56:18.669425 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12753: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:18.669605 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 12753
W20260928 15:56:18.688036 12738 connection.cc:535] client connection to 127.22.157.254:36751 recv error: Network error: failed to read from TLS socket (remote: 127.22.157.254:36751): Connection reset by peer (error 104)
2026-09-28T15:56:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:18 dist-test-slave-1vw0 krb5kdc[12851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:18 dist-test-slave-1vw0 krb5kdc[12851](info): set up 2 sockets
Sep 28 15:56:18 dist-test-slave-1vw0 krb5kdc[12851](info): commencing operation
krb5kdc: starting...
W20260928 15:56:19.841722 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:20 dist-test-slave-1vw0 krb5kdc[12851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:56:20.148689 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:41693
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:41693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:20.492049 12864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:20.492357 12864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:20.492527 12864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:20.500102 12864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:20.500234 12864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:20.500334 12864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:20.500449 12864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:20.500514 12864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:20.510931 12864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:41693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:41693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:20.512341 12864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:20.513787 12864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:20.525844 12872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:20.525906 12869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:20.525938 12870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:20.527194 12864 server_base.cc:1034] running on GCE node
I20260928 15:56:20.528038 12864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:20.529716 12864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:20.530951 12864 hybrid_clock.cc:648] HybridClock initialized: now 1790610980530903 us; error 47 us; skew 500 ppm
I20260928 15:56:20.531317 12864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:20 dist-test-slave-1vw0 krb5kdc[12851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610980, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:20.534534 12864 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:56:20.536149 12864 webserver.cc:460] Webserver started at http://127.22.157.254:36671/ using document root <none> and password file <none>
I20260928 15:56:20.536952 12864 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:20.537101 12864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:20.537421 12864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:20.540414 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "3304c6102fcb4ee4a51d3e05065ed0c2"
format_stamp: "Formatted at 2026-09-28 15:56:20 on dist-test-slave-1vw0"
server_key: "eff64cccef02fc643c8486b44ad6c9f3"
server_key_iv: "b80d7ef63d3fc7945882b374d166d64a"
server_key_version: "encryptionkey@0"
I20260928 15:56:20.541229 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "3304c6102fcb4ee4a51d3e05065ed0c2"
format_stamp: "Formatted at 2026-09-28 15:56:20 on dist-test-slave-1vw0"
server_key: "eff64cccef02fc643c8486b44ad6c9f3"
server_key_iv: "b80d7ef63d3fc7945882b374d166d64a"
server_key_version: "encryptionkey@0"
I20260928 15:56:20.546584 12864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:56:20.550184 12879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:20.551589 12864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:56:20.551791 12864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "3304c6102fcb4ee4a51d3e05065ed0c2"
format_stamp: "Formatted at 2026-09-28 15:56:20 on dist-test-slave-1vw0"
server_key: "eff64cccef02fc643c8486b44ad6c9f3"
server_key_iv: "b80d7ef63d3fc7945882b374d166d64a"
server_key_version: "encryptionkey@0"
I20260928 15:56:20.552002 12864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:20.558061 12864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:20.562124 12864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:20.562440 12864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:20.581624 12864 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:41693
I20260928 15:56:20.581681 12930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:41693 every 8 connection(s)
I20260928 15:56:20.583199 12864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:20.587280 12931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:20.588294 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 12864
I20260928 15:56:20.588480 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:20.589151 23159 external_mini_cluster.cc:1428] Setting key c5dc66e6c528d64e16aeac9e60fce3d9
I20260928 15:56:20.597378 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:20.591053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42299 (local address 127.22.157.254:41693)
0928 15:56:20.591900 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:20.591916 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:20.591957 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:56:20.593797 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:20.593823 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:20.594206 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:20.594548 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:20.595698 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:20.595706 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:20.595719 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:20.595735 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:20.595796 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:20.595819 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:20.595986 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:20.596115 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 15:56:20.596345 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":365,"threads_started":1}
I20260928 15:56:20.601158 12931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2: Bootstrap starting.
I20260928 15:56:20.601209 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:20.605319 12931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:20.606806 12931 log.cc:824] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:20.609457 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:20.606646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49547 (local address 127.22.157.254:41693)
0928 15:56:20.606860 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:20.606866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:20.607397 (+ 531us) server_negotiation.cc:407] Connection header received
0928 15:56:20.607627 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:20.607632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:20.607694 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:20.607802 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:20.608810 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:20.608816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:20.608820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:20.608839 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:20.608885 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:20.608901 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:20.609022 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:20.609117 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 15:56:20.609181 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 15:56:20.610141 12931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2: No bootstrap required, opened a new log
I20260928 15:56:20.618567 12931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3304c6102fcb4ee4a51d3e05065ed0c2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41693 } }
I20260928 15:56:20.619037 12931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:20.619154 12931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3304c6102fcb4ee4a51d3e05065ed0c2, State: Initialized, Role: FOLLOWER
I20260928 15:56:20.619745 12931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3304c6102fcb4ee4a51d3e05065ed0c2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41693 } }
I20260928 15:56:20.619997 12931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:20.620148 12931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:20.620321 12931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:20.622824 12931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3304c6102fcb4ee4a51d3e05065ed0c2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41693 } }
I20260928 15:56:20.623389 12931 leader_election.cc:302] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3304c6102fcb4ee4a51d3e05065ed0c2; no voters:
I20260928 15:56:20.623993 12931 leader_election.cc:288] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:20.624161 12941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:20.624909 12941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 3304c6102fcb4ee4a51d3e05065ed0c2, State: Running, Role: LEADER
I20260928 15:56:20.625474 12941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3304c6102fcb4ee4a51d3e05065ed0c2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41693 } }
I20260928 15:56:20.626294 12931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:20.628973 12942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3304c6102fcb4ee4a51d3e05065ed0c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3304c6102fcb4ee4a51d3e05065ed0c2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41693 } } }
I20260928 15:56:20.629385 12942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:20.630106 12943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3304c6102fcb4ee4a51d3e05065ed0c2. Latest consensus state: current_term: 1 leader_uuid: "3304c6102fcb4ee4a51d3e05065ed0c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3304c6102fcb4ee4a51d3e05065ed0c2" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 41693 } } }
I20260928 15:56:20.630386 12943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:20.631696 12948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:20.637616 12948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:20.646854 12948 catalog_manager.cc:1347] Generated new cluster ID: 608c4bd770bb4ced96bcf895ed8e76d1
I20260928 15:56:20.646979 12948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:20.663966 12948 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:20.664499 12948 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:20.677330 12948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3304c6102fcb4ee4a51d3e05065ed0c2: Generated new TSK 0
I20260928 15:56:20.678463 12948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:20.695972 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12864
W20260928 15:56:20.697698 12926 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42299
W20260928 15:56:20.698487 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:20.698648 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 12864
2026-09-28T15:56:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:20 dist-test-slave-1vw0 krb5kdc[12962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:20 dist-test-slave-1vw0 krb5kdc[12962](info): set up 2 sockets
Sep 28 15:56:20 dist-test-slave-1vw0 krb5kdc[12962](info): commencing operation
krb5kdc: starting...
W20260928 15:56:21.825626 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:22 dist-test-slave-1vw0 krb5kdc[12962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:56:22.136416 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:40027
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:35409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:40027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:22.480914 12975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:22.481251 12975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:22.481552 12975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:22.489660 12975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:22.489789 12975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:22.489917 12975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:22.490042 12975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:22.490136 12975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:22.500468 12975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:35409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:40027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:40027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:22.501947 12975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:22.503394 12975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:22.516776 12980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:22.516848 12981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:22.516764 12983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:22.517988 12975 server_base.cc:1034] running on GCE node
I20260928 15:56:22.518882 12975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:22.520427 12975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:22.521644 12975 hybrid_clock.cc:648] HybridClock initialized: now 1790610982521573 us; error 63 us; skew 500 ppm
I20260928 15:56:22.522050 12975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:22 dist-test-slave-1vw0 krb5kdc[12962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610982, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:22.535123 12975 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:56:22.537201 12975 webserver.cc:460] Webserver started at http://127.22.157.254:34467/ using document root <none> and password file <none>
I20260928 15:56:22.537986 12975 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:22.538146 12975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:22.538497 12975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:22.541893 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "687b0e88030f4b24b306b82d0394094e"
format_stamp: "Formatted at 2026-09-28 15:56:22 on dist-test-slave-1vw0"
server_key: "fb65e5743ff4ae70932bc9741a065ff5"
server_key_iv: "47d48cf75718b57addbf40c7efe0dd85"
server_key_version: "encryptionkey@0"
I20260928 15:56:22.542764 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "687b0e88030f4b24b306b82d0394094e"
format_stamp: "Formatted at 2026-09-28 15:56:22 on dist-test-slave-1vw0"
server_key: "fb65e5743ff4ae70932bc9741a065ff5"
server_key_iv: "47d48cf75718b57addbf40c7efe0dd85"
server_key_version: "encryptionkey@0"
I20260928 15:56:22.548465 12975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 15:56:22.552335 12990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:22.553960 12975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:56:22.554189 12975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "687b0e88030f4b24b306b82d0394094e"
format_stamp: "Formatted at 2026-09-28 15:56:22 on dist-test-slave-1vw0"
server_key: "fb65e5743ff4ae70932bc9741a065ff5"
server_key_iv: "47d48cf75718b57addbf40c7efe0dd85"
server_key_version: "encryptionkey@0"
I20260928 15:56:22.554414 12975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:22.574623 12975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:22.579348 12975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:22.579690 12975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:22.599406 12975 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:40027
I20260928 15:56:22.599511 13041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:40027 every 8 connection(s)
I20260928 15:56:22.600905 12975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:22.605289 13042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:22.606892 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 12975
I20260928 15:56:22.607088 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:22.607633 23159 external_mini_cluster.cc:1428] Setting key d14fcf5e15de845ab901e35e302c75df
I20260928 15:56:22.621645 13042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e: Bootstrap starting.
Sep 28 15:56:22 dist-test-slave-1vw0 krb5kdc[12962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:56:22.625957 13042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:22.627535 13042 log.cc:824] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:22.631265 13042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e: No bootstrap required, opened a new log
I20260928 15:56:22.634073 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:22.609641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43413 (local address 127.22.157.254:40027)
0928 15:56:22.610473 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:22.610485 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:22.610533 (+ 48us) server_negotiation.cc:407] Connection header received
0928 15:56:22.612203 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:22.612234 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:22.612705 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:22.613168 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:22.615304 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:22.616917 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:22.619031 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:22.619990 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:22.620191 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:22.625496 (+ 5305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:22.625506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:22.625524 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:22.625574 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:22.629347 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:22.630237 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:22.630245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:22.630252 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:22.630448 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:22.630892 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:22.630898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:22.630901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:22.631544 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:22.631888 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:22.632378 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 15:56:22.632917 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":411,"threads_started":1}
I20260928 15:56:22.638172 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:22.640081 13042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b0e88030f4b24b306b82d0394094e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40027 } }
I20260928 15:56:22.640529 13042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:22.640661 13042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687b0e88030f4b24b306b82d0394094e, State: Initialized, Role: FOLLOWER
I20260928 15:56:22.641409 13042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b0e88030f4b24b306b82d0394094e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40027 } }
I20260928 15:56:22.641672 13042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:22.641839 13042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:22.642033 13042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:22.644960 13042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b0e88030f4b24b306b82d0394094e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40027 } }
I20260928 15:56:22.645545 13042 leader_election.cc:302] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 687b0e88030f4b24b306b82d0394094e; no voters:
I20260928 15:56:22.646234 13042 leader_election.cc:288] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:22.646353 13052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:22.648765 13042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:22.649127 13052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [term 1 LEADER]: Becoming Leader. State: Replica: 687b0e88030f4b24b306b82d0394094e, State: Running, Role: LEADER
I20260928 15:56:22.649806 13052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b0e88030f4b24b306b82d0394094e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40027 } }
I20260928 15:56:22.654013 13053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "687b0e88030f4b24b306b82d0394094e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b0e88030f4b24b306b82d0394094e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40027 } } }
I20260928 15:56:22.654446 13053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:22.660542 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:22.642952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42341 (local address 127.22.157.254:40027)
0928 15:56:22.643177 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:22.643183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:22.643594 (+ 411us) server_negotiation.cc:407] Connection header received
0928 15:56:22.643745 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:22.643750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:22.643851 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:22.643974 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:22.645589 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:22.646824 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:22.648651 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:22.649435 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:22.649580 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:22.652695 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:22.652703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:22.652709 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:22.652779 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:22.657378 (+ 4599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:22.658282 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:22.658288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:22.658291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:22.658437 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:22.658845 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:22.658850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:22.658853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:22.659334 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:22.659622 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:22.659961 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 15:56:22.660191 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:56:22.662531 13058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:22.665066 13054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687b0e88030f4b24b306b82d0394094e. Latest consensus state: current_term: 1 leader_uuid: "687b0e88030f4b24b306b82d0394094e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687b0e88030f4b24b306b82d0394094e" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 40027 } } }
I20260928 15:56:22.665534 13054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:22.670249 13058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:22.680001 13058 catalog_manager.cc:1347] Generated new cluster ID: df1ff66cf567429ba6a4e6ecc860f35c
I20260928 15:56:22.680127 13058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:22.690323 13058 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:22.691954 13058 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:22.699321 13058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 687b0e88030f4b24b306b82d0394094e: Generated new TSK 0
I20260928 15:56:22.700225 13058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:22.705711 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 12975
I20260928 15:56:22.706856 13037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:22.799942 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 12975
2026-09-28T15:56:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2101 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:22 dist-test-slave-1vw0 krb5kdc[13074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:22 dist-test-slave-1vw0 krb5kdc[13074](info): set up 2 sockets
Sep 28 15:56:22 dist-test-slave-1vw0 krb5kdc[13074](info): commencing operation
krb5kdc: starting...
W20260928 15:56:23.933804 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:24 dist-test-slave-1vw0 krb5kdc[13074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:56:24.255487 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43479
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:43479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:24.609246 13087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:24.609655 13087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:24.609834 13087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:24.617978 13087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:24.618184 13087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:24.618309 13087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:24.618430 13087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:24.618513 13087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:24.629190 13087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:43479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:24.630667 13087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:24.632092 13087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:24.644867 13092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:24.644783 13093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:24.644848 13095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:24.645792 13087 server_base.cc:1034] running on GCE node
I20260928 15:56:24.646667 13087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:24.648212 13087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:24.649453 13087 hybrid_clock.cc:648] HybridClock initialized: now 1790610984649410 us; error 54 us; skew 500 ppm
I20260928 15:56:24.649885 13087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:24 dist-test-slave-1vw0 krb5kdc[13074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610984, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:24.653228 13087 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:56:24.654887 13087 webserver.cc:460] Webserver started at http://127.22.157.254:36989/ using document root <none> and password file <none>
I20260928 15:56:24.655632 13087 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:24.655783 13087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:24.656103 13087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:24.659246 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "21a5278d4ed641c7994e06eb5a73e204"
format_stamp: "Formatted at 2026-09-28 15:56:24 on dist-test-slave-1vw0"
server_key: "690d57e938d9d610b6450989292a49f7"
server_key_iv: "629eeff3b4ac028202e84d5f5e63e3af"
server_key_version: "encryptionkey@0"
I20260928 15:56:24.660033 13087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "21a5278d4ed641c7994e06eb5a73e204"
format_stamp: "Formatted at 2026-09-28 15:56:24 on dist-test-slave-1vw0"
server_key: "690d57e938d9d610b6450989292a49f7"
server_key_iv: "629eeff3b4ac028202e84d5f5e63e3af"
server_key_version: "encryptionkey@0"
I20260928 15:56:24.665625 13087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 15:56:24.669361 13102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:24.670940 13087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:56:24.671175 13087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "21a5278d4ed641c7994e06eb5a73e204"
format_stamp: "Formatted at 2026-09-28 15:56:24 on dist-test-slave-1vw0"
server_key: "690d57e938d9d610b6450989292a49f7"
server_key_iv: "629eeff3b4ac028202e84d5f5e63e3af"
server_key_version: "encryptionkey@0"
I20260928 15:56:24.671408 13087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:24.722678 13087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:24.727200 13087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:24.727516 13087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:24.746961 13087 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:43479
I20260928 15:56:24.746981 13153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:43479 every 8 connection(s)
I20260928 15:56:24.748576 13087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:24.752413 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:24.756059 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 13087
I20260928 15:56:24.756263 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:24.756814 23159 external_mini_cluster.cc:1428] Setting key 43277dc312f3fc3a9c6f23a3030063dd
I20260928 15:56:24.764887 13154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204: Bootstrap starting.
I20260928 15:56:24.768250 13154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:24.769534 13154 log.cc:824] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:24.770728 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:24.758891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54539 (local address 127.22.157.254:43479)
0928 15:56:24.759664 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:24.759676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:24.759718 (+ 42us) server_negotiation.cc:407] Connection header received
0928 15:56:24.761493 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:24.761516 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:24.761955 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:24.762371 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:24.762952 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:24.764245 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:24.765716 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:24.766465 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:24.766683 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:24.767968 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:24.767975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:24.767991 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:24.768631 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:24.768718 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:24.768805 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:24.768925 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:24.769178 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 15:56:24.769439 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":342,"threads_started":1}
I20260928 15:56:24.772487 13154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204: No bootstrap required, opened a new log
I20260928 15:56:24.774562 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:24.781348 13154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a5278d4ed641c7994e06eb5a73e204" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43479 } }
I20260928 15:56:24.781802 13154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:24.781924 13154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a5278d4ed641c7994e06eb5a73e204, State: Initialized, Role: FOLLOWER
I20260928 15:56:24.782620 13154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a5278d4ed641c7994e06eb5a73e204" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43479 } }
I20260928 15:56:24.782927 13154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:24.783062 13154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:24.783221 13154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:24.786106 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:24.779160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41623 (local address 127.22.157.254:43479)
0928 15:56:24.779375 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:24.779381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:24.779948 (+ 567us) server_negotiation.cc:407] Connection header received
0928 15:56:24.780179 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:24.780182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:24.780251 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:24.780365 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:24.780994 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:24.781703 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:24.783634 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:24.784096 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:24.784214 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:24.785410 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:24.785414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:24.785417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:24.785435 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:24.785470 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:24.785485 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:24.785554 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:24.785709 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 15:56:24.785780 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 15:56:24.786360 13154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a5278d4ed641c7994e06eb5a73e204" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43479 } }
I20260928 15:56:24.786923 13154 leader_election.cc:302] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21a5278d4ed641c7994e06eb5a73e204; no voters:
I20260928 15:56:24.787606 13154 leader_election.cc:288] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:24.787717 13164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:24.788424 13164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [term 1 LEADER]: Becoming Leader. State: Replica: 21a5278d4ed641c7994e06eb5a73e204, State: Running, Role: LEADER
I20260928 15:56:24.789083 13164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a5278d4ed641c7994e06eb5a73e204" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43479 } }
I20260928 15:56:24.790165 13154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:24.792260 13165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21a5278d4ed641c7994e06eb5a73e204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a5278d4ed641c7994e06eb5a73e204" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43479 } } }
I20260928 15:56:24.792661 13165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:24.793562 13170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:24.792392 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21a5278d4ed641c7994e06eb5a73e204. Latest consensus state: current_term: 1 leader_uuid: "21a5278d4ed641c7994e06eb5a73e204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a5278d4ed641c7994e06eb5a73e204" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43479 } } }
I20260928 15:56:24.793905 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204 [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:24.799870 13170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:24.809747 13170 catalog_manager.cc:1347] Generated new cluster ID: d4ecb04d62884331b7a8ec1d4c5a4450
I20260928 15:56:24.809890 13170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:24.822809 13170 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:24.824172 13170 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:24.834160 13170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21a5278d4ed641c7994e06eb5a73e204: Generated new TSK 0
I20260928 15:56:24.835103 13170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:24.847776 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13087
W20260928 15:56:24.848950 13149 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54539
W20260928 15:56:24.849674 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13087: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:24.849828 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 13087
2026-09-28T15:56:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2049 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:24 dist-test-slave-1vw0 krb5kdc[13185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:24 dist-test-slave-1vw0 krb5kdc[13185](info): set up 2 sockets
Sep 28 15:56:24 dist-test-slave-1vw0 krb5kdc[13185](info): commencing operation
krb5kdc: starting...
W20260928 15:56:25.973771 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:26 dist-test-slave-1vw0 krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:56:26.280532 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45485
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:45485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:26.625695 13198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:26.626019 13198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:26.626219 13198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:26.633814 13198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:26.633941 13198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:26.634039 13198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:26.634143 13198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:26.634223 13198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:26.644459 13198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:45485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:45485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:26.645921 13198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:26.647308 13198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:26.659708 13203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:26.659593 13206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:26.660145 13204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:26.661371 13198 server_base.cc:1034] running on GCE node
I20260928 15:56:26.662354 13198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:26.663890 13198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:26.665144 13198 hybrid_clock.cc:648] HybridClock initialized: now 1790610986665110 us; error 62 us; skew 500 ppm
I20260928 15:56:26.665530 13198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:26 dist-test-slave-1vw0 krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610986, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:26.668889 13198 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:56:26.670719 13198 webserver.cc:460] Webserver started at http://127.22.157.254:33097/ using document root <none> and password file <none>
I20260928 15:56:26.671499 13198 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:26.671658 13198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:26.671981 13198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:26.675303 13198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "16cb8ade05564153bc93f6a8c175f38b"
format_stamp: "Formatted at 2026-09-28 15:56:26 on dist-test-slave-1vw0"
server_key: "661eedb90aaf21efdd117d836838463b"
server_key_iv: "52ab7315acd0fb46845e78a17ee361d4"
server_key_version: "encryptionkey@0"
I20260928 15:56:26.676087 13198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "16cb8ade05564153bc93f6a8c175f38b"
format_stamp: "Formatted at 2026-09-28 15:56:26 on dist-test-slave-1vw0"
server_key: "661eedb90aaf21efdd117d836838463b"
server_key_iv: "52ab7315acd0fb46845e78a17ee361d4"
server_key_version: "encryptionkey@0"
I20260928 15:56:26.681803 13198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:56:26.685547 13213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:26.687150 13198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:56:26.687345 13198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "16cb8ade05564153bc93f6a8c175f38b"
format_stamp: "Formatted at 2026-09-28 15:56:26 on dist-test-slave-1vw0"
server_key: "661eedb90aaf21efdd117d836838463b"
server_key_iv: "52ab7315acd0fb46845e78a17ee361d4"
server_key_version: "encryptionkey@0"
I20260928 15:56:26.687564 13198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:26.693815 13198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:26.697849 13198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:26.698163 13198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:26.717968 13198 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:45485
I20260928 15:56:26.717999 13264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:45485 every 8 connection(s)
I20260928 15:56:26.719518 13198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:26.720696 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 13198
I20260928 15:56:26.720911 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:26.721467 23159 external_mini_cluster.cc:1428] Setting key 4c34c79320850bc5f73b57a942126c11
I20260928 15:56:26.724992 13265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:26.729707 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:26.723438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46675 (local address 127.22.157.254:45485)
0928 15:56:26.724229 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:26.724242 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:26.724283 (+ 41us) server_negotiation.cc:407] Connection header received
0928 15:56:26.726155 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:26.726179 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:26.726516 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:26.726933 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:26.728021 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:26.728027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:26.728043 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:26.728058 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:26.728137 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:26.728161 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:26.728395 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:26.728529 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 15:56:26.728778 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":329,"threads_started":1}
I20260928 15:56:26.733522 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:26.736775 13265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b: Bootstrap starting.
I20260928 15:56:26.741110 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:26.738085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56875 (local address 127.22.157.254:45485)
0928 15:56:26.738346 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:26.738352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:26.738895 (+ 543us) server_negotiation.cc:407] Connection header received
0928 15:56:26.739101 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:26.739106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:26.739166 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:26.739282 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:26.740410 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:26.740416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:26.740421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 15:56:26.740442 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:26.740494 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 15:56:26.740510 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:26.740649 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:26.740759 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 15:56:26.740825 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 15:56:26.741113 13265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:26.742553 13265 log.cc:824] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:26.745823 13265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b: No bootstrap required, opened a new log
I20260928 15:56:26.753679 13265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cb8ade05564153bc93f6a8c175f38b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45485 } }
I20260928 15:56:26.754069 13265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 15:56:26.754151 13265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16cb8ade05564153bc93f6a8c175f38b, State: Initialized, Role: FOLLOWER
I20260928 15:56:26.754714 13265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cb8ade05564153bc93f6a8c175f38b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45485 } }
I20260928 15:56:26.754911 13265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 15:56:26.755013 13265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 15:56:26.755139 13265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:26.757695 13265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cb8ade05564153bc93f6a8c175f38b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45485 } }
I20260928 15:56:26.758208 13265 leader_election.cc:302] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16cb8ade05564153bc93f6a8c175f38b; no voters:
I20260928 15:56:26.758810 13265 leader_election.cc:288] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 15:56:26.758955 13275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:26.759745 13275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [term 1 LEADER]: Becoming Leader. State: Replica: 16cb8ade05564153bc93f6a8c175f38b, State: Running, Role: LEADER
I20260928 15:56:26.760381 13275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cb8ade05564153bc93f6a8c175f38b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45485 } }
I20260928 15:56:26.761224 13265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:26.763691 13277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16cb8ade05564153bc93f6a8c175f38b. Latest consensus state: current_term: 1 leader_uuid: "16cb8ade05564153bc93f6a8c175f38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cb8ade05564153bc93f6a8c175f38b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45485 } } }
I20260928 15:56:26.764213 13277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:26.763692 13276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16cb8ade05564153bc93f6a8c175f38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16cb8ade05564153bc93f6a8c175f38b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 45485 } } }
I20260928 15:56:26.764623 13276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:26.765277 13283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:26.770694 13283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:26.783082 13283 catalog_manager.cc:1347] Generated new cluster ID: 1ba45987210049c4ae948b99c11cfc1c
I20260928 15:56:26.783241 13283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:26.804989 13283 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:26.805665 13283 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:26.816322 13283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16cb8ade05564153bc93f6a8c175f38b: Generated new TSK 0
I20260928 15:56:26.817449 13283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:26.837237 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13198
W20260928 15:56:26.838583 13260 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46675
W20260928 15:56:26.839668 23159 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13198: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 15:56:26.839838 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 13198
2026-09-28T15:56:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1989 ms)
[----------] 18 tests from AuthTokenIssuingTest (37162 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:26 dist-test-slave-1vw0 krb5kdc[13296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:26 dist-test-slave-1vw0 krb5kdc[13296](info): set up 2 sockets
Sep 28 15:56:26 dist-test-slave-1vw0 krb5kdc[13296](info): commencing operation
krb5kdc: starting...
W20260928 15:56:27.989813 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:28 dist-test-slave-1vw0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:56:28.304098 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43865
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:43609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:43865,127.22.157.253:35627,127.22.157.252:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:28.652089 13309 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:56:28.652509 13309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:28.652648 13309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:28.652829 13309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:28.660429 13309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:28.660554 13309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:28.660655 13309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:28.660760 13309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:28.660868 13309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:28.671388 13309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:43609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:43865,127.22.157.253:35627,127.22.157.252:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:43865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:28.672921 13309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:28.674417 13309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:28.687686 13317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:28.687732 13315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:28.687757 13314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:28.688952 13309 server_base.cc:1034] running on GCE node
I20260928 15:56:28.689802 13309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:28.691380 13309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:28.692657 13309 hybrid_clock.cc:648] HybridClock initialized: now 1790610988692534 us; error 57 us; skew 500 ppm
I20260928 15:56:28.693071 13309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:28 dist-test-slave-1vw0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610988, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:28.696662 13309 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:56:28.698406 13309 webserver.cc:460] Webserver started at http://127.22.157.254:46219/ using document root <none> and password file <none>
I20260928 15:56:28.699250 13309 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:28.699410 13309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:28.699805 13309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:28.703074 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "d6ac2ceb368344cf977627952ec09b1b"
format_stamp: "Formatted at 2026-09-28 15:56:28 on dist-test-slave-1vw0"
server_key: "e3a0d84c2319d5629b4a6d075a477382"
server_key_iv: "1c51b214e4fb1db5ee8ef0c37ec738c8"
server_key_version: "encryptionkey@0"
I20260928 15:56:28.703953 13309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "d6ac2ceb368344cf977627952ec09b1b"
format_stamp: "Formatted at 2026-09-28 15:56:28 on dist-test-slave-1vw0"
server_key: "e3a0d84c2319d5629b4a6d075a477382"
server_key_iv: "1c51b214e4fb1db5ee8ef0c37ec738c8"
server_key_version: "encryptionkey@0"
I20260928 15:56:28.709954 13309 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 15:56:28.713953 13324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:28.715653 13309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 15:56:28.715898 13309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "d6ac2ceb368344cf977627952ec09b1b"
format_stamp: "Formatted at 2026-09-28 15:56:28 on dist-test-slave-1vw0"
server_key: "e3a0d84c2319d5629b4a6d075a477382"
server_key_iv: "1c51b214e4fb1db5ee8ef0c37ec738c8"
server_key_version: "encryptionkey@0"
I20260928 15:56:28.716182 13309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:28.745841 13309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:28.750700 13309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:28.751039 13309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:28.771929 13309 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:43865
I20260928 15:56:28.772023 13375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:43865 every 8 connection(s)
I20260928 15:56:28.773522 13309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:28.775314 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 13309
I20260928 15:56:28.775592 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:28.776381 23159 external_mini_cluster.cc:1428] Setting key c98af2660933ff48b160472d706d59a8
I20260928 15:56:28.778249 13376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:28.785668 13376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:28 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610988, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.253@KRBTEST.COM" created.
I20260928 15:56:28.815524 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:28.789421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43865 (local address 127.0.0.1:34764)
0928 15:56:28.790498 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:28.790561 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:28.793917 (+ 3356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:28.795193 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:28.795204 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:28.796176 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:28.796869 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:28.796906 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:28.798741 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:28.798750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:28.800921 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:28.800944 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:28.802089 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:28.802099 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:28.802347 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:28.803286 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:28.803319 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:28.806744 (+ 3425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:28.810554 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:28.810569 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:28.810584 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:28.811056 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:28.811528 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:28.811535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:28.811540 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:28.811705 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:28.812586 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:28.812596 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:28.812909 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 15:56:28.813183 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 15:56:28.813981 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":255,"thread_start_us":444,"threads_started":1}
I20260928 15:56:28.815575 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:28.789421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.22.157.254:43865)
0928 15:56:28.791728 (+ 2307us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:28.791739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:28.791792 (+ 53us) server_negotiation.cc:407] Connection header received
0928 15:56:28.794534 (+ 2742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:28.794541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:28.794877 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:28.795069 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:28.797182 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:28.798532 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:28.801114 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:28.801790 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:28.801951 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:28.807090 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:28.807100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:28.807115 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:28.807156 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:28.810354 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:28.811222 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:28.811229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:28.811231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:28.811398 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:28.811825 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:28.811830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:28.811833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:28.812425 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:28.812658 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:28.813327 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 15:56:28.813849 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":2081,"spinlock_wait_cycles":28544,"thread_start_us":1784,"threads_started":1}
I20260928 15:56:28.819587 13376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:56:28.821494 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:28.820689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33652)
0928 15:56:28.821088 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:28.821255 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237}
W20260928 15:56:28.821923 13325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
I20260928 15:56:28.823836 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:28.823369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33654)
0928 15:56:28.823577 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:28.823671 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260928 15:56:28.824182 13376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:35627: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab.
Entry for principal kudu/127.22.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab.
I20260928 15:56:28.849182 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.253:35627
--webserver_interface=127.22.157.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab
--principal=kudu/127.22.157.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:43609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:43865,127.22.157.253:35627,127.22.157.252:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:56:28.873667 13376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } attempt: 1
I20260928 15:56:28.875101 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:28.874539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33656)
0928 15:56:28.874781 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:28.874947 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 15:56:28.876540 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:28.876030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33666)
0928 15:56:28.876212 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:28.876355 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 15:56:28.876854 13376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:35627: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
I20260928 15:56:28.945190 13376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } attempt: 2
I20260928 15:56:28.946439 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:28.945922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33676)
0928 15:56:28.946209 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:28.946300 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 15:56:28.947805 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:28.947331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33692)
0928 15:56:28.947525 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:28.947624 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 15:56:28.948045 13376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:35627: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
I20260928 15:56:29.039366 13376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } attempt: 3
I20260928 15:56:29.040843 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.040201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33708)
0928 15:56:29.040484 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.040601 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 15:56:29.042081 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.041631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33722)
0928 15:56:29.041812 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.041904 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 15:56:29.042321 13376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:35627: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
I20260928 15:56:29.185580 13376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } attempt: 4
I20260928 15:56:29.186895 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.186262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33734)
0928 15:56:29.186617 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.186722 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260928 15:56:29.188181 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.187728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33748)
0928 15:56:29.187896 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.188018 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 15:56:29.188427 13376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:35627: Network error: Client connection negotiation failed: client connection to 127.22.157.253:35627: connect: Connection refused (error 111)
W20260928 15:56:29.205081 13383 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:56:29.205454 13383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:29.205583 13383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:29.205731 13383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:29.213594 13383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:29.213739 13383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:29.213840 13383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:29.213925 13383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:29.214030 13383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:29.224292 13383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:43609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:43865,127.22.157.253:35627,127.22.157.252:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab
--principal=kudu/127.22.157.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.253:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.157.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:29.225631 13383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:29.226964 13383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:29.239146 13388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:29.239271 13389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:29.239553 13391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:29.239981 13383 server_base.cc:1034] running on GCE node
I20260928 15:56:29.240878 13383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:29.242316 13383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:29.243570 13383 hybrid_clock.cc:648] HybridClock initialized: now 1790610989243517 us; error 44 us; skew 500 ppm
I20260928 15:56:29.243911 13383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610989, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:29.247143 13383 init.cc:377] Logged in from keytab as kudu/127.22.157.253@KRBTEST.COM (short username kudu)
I20260928 15:56:29.248812 13383 webserver.cc:460] Webserver started at http://127.22.157.253:33275/ using document root <none> and password file <none>
I20260928 15:56:29.249504 13383 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:29.249614 13383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:29.249953 13383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:29.252908 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/data/instance:
uuid: "f5fe5e7d599b4be9a5afec403cdad988"
format_stamp: "Formatted at 2026-09-28 15:56:29 on dist-test-slave-1vw0"
server_key: "04d794c893c5cec97a41bf416facb303"
server_key_iv: "b5f936b2fca4daa749c289da21fa9098"
server_key_version: "encryptionkey@0"
I20260928 15:56:29.253733 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/wal/instance:
uuid: "f5fe5e7d599b4be9a5afec403cdad988"
format_stamp: "Formatted at 2026-09-28 15:56:29 on dist-test-slave-1vw0"
server_key: "04d794c893c5cec97a41bf416facb303"
server_key_iv: "b5f936b2fca4daa749c289da21fa9098"
server_key_version: "encryptionkey@0"
I20260928 15:56:29.258916 13383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 15:56:29.262626 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:29.264022 13383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:56:29.264222 13383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/wal
uuid: "f5fe5e7d599b4be9a5afec403cdad988"
format_stamp: "Formatted at 2026-09-28 15:56:29 on dist-test-slave-1vw0"
server_key: "04d794c893c5cec97a41bf416facb303"
server_key_iv: "b5f936b2fca4daa749c289da21fa9098"
server_key_version: "encryptionkey@0"
I20260928 15:56:29.264461 13383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:29.275712 13383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:29.279907 13383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:29.280190 13383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:29.299628 13383 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.253:35627
I20260928 15:56:29.299638 13449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.253:35627 every 8 connection(s)
I20260928 15:56:29.301126 13383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
I20260928 15:56:29.304967 13450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:29.310695 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 13383
I20260928 15:56:29.310901 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-1/wal/instance
I20260928 15:56:29.311458 23159 external_mini_cluster.cc:1428] Setting key 2efdbee2b9efe4e3506b956b45869929
I20260928 15:56:29.311933 13450 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610989, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:56:29.335908 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.315492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34766 (local address 127.22.157.254:43865)
0928 15:56:29.315722 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.315729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.317104 (+ 1375us) server_negotiation.cc:407] Connection header received
0928 15:56:29.319080 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.319088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.319191 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.319318 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.320982 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.322108 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.323702 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.324112 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.324251 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.328510 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.328519 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.328527 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.328576 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.332366 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.333267 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.333276 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.333283 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.333439 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.333858 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.333866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.333872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.334344 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.334499 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.335224 (+ 725us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.335489 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 15:56:29.336655 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.316063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43865 (local address 127.0.0.1:34766)
0928 15:56:29.316922 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.316975 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:29.318790 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:29.319508 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:29.319516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:29.320145 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:29.320778 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.320807 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.322290 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.322294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.323505 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.323517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.324352 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.324359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.324696 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.325480 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:29.325516 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:29.328247 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:29.332518 (+ 4271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.332527 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.332541 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.333117 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.333550 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.333554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.333556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.333738 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.334466 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:29.334472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:29.334815 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 15:56:29.335045 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 15:56:29.335463 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":383,"threads_started":1}
I20260928 15:56:29.338603 13450 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.22.157.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.252@KRBTEST.COM" created.
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610989, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
I20260928 15:56:29.354988 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.339485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33762)
0928 15:56:29.339728 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.339751 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:29.339877 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:29.340912 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:29.340918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:29.341451 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:29.341671 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.341685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.343288 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.343293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.344241 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.344253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.345296 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.345301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.345439 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.346111 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:29.346129 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:29.348139 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:29.352460 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.352464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.352467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.352917 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.353327 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.353330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.353333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.353474 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.354258 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:29.354262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:29.354395 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 15:56:29.354504 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 15:56:29.354722 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 15:56:29.355497 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.339709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.22.157.253:35627)
0928 15:56:29.340436 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.340442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.340468 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:56:29.340516 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.340522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.340701 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.340883 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.341880 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.343130 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.344382 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.345137 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.345245 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.348369 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.348387 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.348408 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.348453 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.352323 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.353043 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.353055 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.353061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.353208 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.353575 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.353582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.353594 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.354089 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.354332 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.354709 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.355010 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":418,"threads_started":1}
I20260928 15:56:29.358000 13450 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:56:29.359437 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.358868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53016)
0928 15:56:29.359073 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.359230 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 15:56:29.359929 13400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
I20260928 15:56:29.361807 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.361351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53024)
0928 15:56:29.361527 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.361603 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 15:56:29.362052 13450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:44971: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab.
Entry for principal kudu/127.22.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab.
I20260928 15:56:29.386647 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.252:44971
--webserver_interface=127.22.157.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab
--principal=kudu/127.22.157.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:43609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:43865,127.22.157.253:35627,127.22.157.252:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:56:29.411363 13450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } attempt: 1
I20260928 15:56:29.412668 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.412198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53026)
0928 15:56:29.412410 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.412515 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 15:56:29.413877 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.413457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53042)
0928 15:56:29.413686 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.413769 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260928 15:56:29.414076 13450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:44971: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
I20260928 15:56:29.482297 13450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } attempt: 2
I20260928 15:56:29.483711 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.483145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53052)
0928 15:56:29.483397 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.483512 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 15:56:29.484861 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.484431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53060)
0928 15:56:29.484593 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.484683 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 15:56:29.485081 13450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:44971: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
I20260928 15:56:29.487736 13376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } attempt: 5
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610988, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
I20260928 15:56:29.502821 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.488477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.22.157.253:35627)
0928 15:56:29.488663 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.488671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.488699 (+ 28us) server_negotiation.cc:407] Connection header received
0928 15:56:29.488925 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.488933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.489060 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.489203 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.489936 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.491077 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.492212 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.492885 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.493029 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.496336 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.496342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.496347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.496389 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.500085 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.500829 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.500836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.500844 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.500970 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.501377 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.501384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.501387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.501696 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.501852 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.502227 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.502455 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:56:29.502821 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.488412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33768)
0928 15:56:29.488589 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.488608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:29.488767 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:29.489234 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:29.489237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:29.489605 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:29.489781 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.489789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.491209 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.491213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.492060 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.492070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.493030 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.493034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.493214 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.493840 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:29.493857 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:29.496105 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:29.500259 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.500263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.500266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.500659 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.501096 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.501101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.501105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.501265 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.501818 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:29.501823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:29.501978 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 15:56:29.502125 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 15:56:29.502473 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 15:56:29.503940 13376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:56:29.505393 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.504796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53064)
0928 15:56:29.505038 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.505170 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 15:56:29.506464 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.506005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53068)
0928 15:56:29.506200 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.506297 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 15:56:29.506726 13376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:44971: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
I20260928 15:56:29.558017 13376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } attempt: 1
I20260928 15:56:29.559355 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.558741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53070)
0928 15:56:29.559025 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.559148 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 15:56:29.560463 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.559993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53076)
0928 15:56:29.560195 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.560296 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 15:56:29.560710 13376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:44971: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
I20260928 15:56:29.576435 13450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } attempt: 3
I20260928 15:56:29.577798 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.577235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53092)
0928 15:56:29.577436 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.577565 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 15:56:29.578936 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.578497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53096)
0928 15:56:29.578726 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.578813 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260928 15:56:29.579123 13450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:44971: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
I20260928 15:56:29.629019 13376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } attempt: 2
I20260928 15:56:29.630417 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.629796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53100)
0928 15:56:29.630071 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.630199 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 15:56:29.631692 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.631243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53102)
0928 15:56:29.631442 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.631537 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 15:56:29.631961 13376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:44971: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
I20260928 15:56:29.722323 13450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } attempt: 4
I20260928 15:56:29.723569 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.722953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53112)
0928 15:56:29.723246 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.723363 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 15:56:29.724937 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.724438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53126)
0928 15:56:29.724653 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.724769 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 15:56:29.725231 13450 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:44971: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
I20260928 15:56:29.738206 13376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } attempt: 3
I20260928 15:56:29.739516 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.738927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53134)
0928 15:56:29.739188 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.739300 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 15:56:29.740653 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.740215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53140)
0928 15:56:29.740432 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.740524 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260928 15:56:29.740908 13376 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:44971: Network error: Client connection negotiation failed: client connection to 127.22.157.252:44971: connect: Connection refused (error 111)
W20260928 15:56:29.759013 13457 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:56:29.759353 13457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:29.759425 13457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:29.759507 13457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:29.767246 13457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:29.767347 13457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:29.767448 13457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:29.767542 13457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:29.767653 13457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:29.779204 13457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:43609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:43865,127.22.157.253:35627,127.22.157.252:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab
--principal=kudu/127.22.157.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.252:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.157.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:29.781065 13457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:29.783219 13457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:29.797173 13463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:29.797158 13465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:29.797883 13462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:29.798278 13457 server_base.cc:1034] running on GCE node
I20260928 15:56:29.799137 13457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:29.800614 13457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:29.801888 13457 hybrid_clock.cc:648] HybridClock initialized: now 1790610989801825 us; error 34 us; skew 500 ppm
I20260928 15:56:29.802263 13457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610989, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:29.805536 13457 init.cc:377] Logged in from keytab as kudu/127.22.157.252@KRBTEST.COM (short username kudu)
I20260928 15:56:29.807098 13457 webserver.cc:460] Webserver started at http://127.22.157.252:44671/ using document root <none> and password file <none>
I20260928 15:56:29.807762 13457 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:29.807849 13457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:29.808125 13457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:29.811218 13457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/data/instance:
uuid: "d6e81b1f54eb4a71abec5aac0321184a"
format_stamp: "Formatted at 2026-09-28 15:56:29 on dist-test-slave-1vw0"
server_key: "149eb4016b6e3fe93df6a2ae9db63ff3"
server_key_iv: "86f279986d237e3b3d5944698935d038"
server_key_version: "encryptionkey@0"
I20260928 15:56:29.811962 13457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/wal/instance:
uuid: "d6e81b1f54eb4a71abec5aac0321184a"
format_stamp: "Formatted at 2026-09-28 15:56:29 on dist-test-slave-1vw0"
server_key: "149eb4016b6e3fe93df6a2ae9db63ff3"
server_key_iv: "86f279986d237e3b3d5944698935d038"
server_key_version: "encryptionkey@0"
I20260928 15:56:29.817284 13457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 15:56:29.820744 13472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:29.822191 13457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 15:56:29.822330 13457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/wal
uuid: "d6e81b1f54eb4a71abec5aac0321184a"
format_stamp: "Formatted at 2026-09-28 15:56:29 on dist-test-slave-1vw0"
server_key: "149eb4016b6e3fe93df6a2ae9db63ff3"
server_key_iv: "86f279986d237e3b3d5944698935d038"
server_key_version: "encryptionkey@0"
I20260928 15:56:29.822513 13457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:29.837589 13457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:29.841631 13457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:29.841856 13457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:29.860298 13457 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.252:44971
I20260928 15:56:29.860318 13523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.252:44971 every 8 connection(s)
I20260928 15:56:29.861807 13457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
I20260928 15:56:29.865936 13524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:29.868120 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 13457
I20260928 15:56:29.868396 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790610618730370-23159-0/minicluster-data/master-2/wal/instance
I20260928 15:56:29.869029 23159 external_mini_cluster.cc:1428] Setting key 3eb49e2b414415c317dc8884b79c15d9
I20260928 15:56:29.874765 13524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610989, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:56:29.892985 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.877770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.22.157.254:43865)
0928 15:56:29.878361 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.878368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.878825 (+ 457us) server_negotiation.cc:407] Connection header received
0928 15:56:29.879006 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.879011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.879114 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.879236 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.880272 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.881153 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.882453 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.883157 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.883321 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.887254 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.887259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.887262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.887292 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.889979 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.890733 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.890739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.890743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.890902 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.891317 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.891325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.891331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.891666 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.891816 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.892305 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.892535 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":249,"threads_started":1}
I20260928 15:56:29.893347 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.871201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.22.157.252:44971)
0928 15:56:29.872126 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.872143 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.872195 (+ 52us) server_negotiation.cc:407] Connection header received
0928 15:56:29.874567 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.874595 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.875150 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.875619 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.877048 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.878981 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.880315 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.881044 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.881176 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.885789 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.885803 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.885819 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.885855 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.888865 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.889568 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.889577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.889596 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.889737 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.890137 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.890142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.890146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.890786 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.891087 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.891560 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.892005 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":399,"threads_started":1}
I20260928 15:56:29.893396 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.877813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:43865 (local address 127.0.0.1:34778)
0928 15:56:29.878681 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.878729 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:29.878875 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:29.879290 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:29.879294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:29.879880 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:29.880117 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.880125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.881315 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.881320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.882313 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.882324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.883421 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.883429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.883625 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.884281 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:29.884309 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:29.887029 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:29.890101 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.890113 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.890123 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.890614 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.891015 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.891020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.891024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.891191 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.891846 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:29.891854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:29.892011 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 15:56:29.892166 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 15:56:29.892393 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":459,"threads_started":1}
I20260928 15:56:29.895085 13524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610989, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:56:29.910157 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.895770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:35627 (local address 127.0.0.1:33784)
0928 15:56:29.895967 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.895987 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:29.896108 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:29.896482 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:29.896487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:29.897017 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:29.897271 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.897283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.898235 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.898239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.899129 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.899140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.899924 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.899930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.900123 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.900784 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:29.900820 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:29.903085 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:29.907111 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.907117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.907122 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.907708 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.908259 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.908263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.908265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.908410 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.909245 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:29.909249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:29.909399 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 15:56:29.909514 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 15:56:29.909738 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 15:56:29.910392 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.895995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.22.157.253:35627)
0928 15:56:29.896206 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.896212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.896236 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:56:29.896282 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.896286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.896350 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.896460 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.897421 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.898089 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.899293 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.899767 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.899904 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.903304 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.903312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.903318 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.903366 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.906939 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.907844 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.907850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.907854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.908107 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.908582 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.908587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.908591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.909077 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.909267 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.909657 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.909920 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:56:29.911392 13524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:56:29.914844 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.896865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.22.157.254:43865)
0928 15:56:29.897035 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.897040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.897058 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:56:29.897211 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.897217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.897299 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.897390 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.899382 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.900139 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.901581 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.902116 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.902251 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.907869 (+ 5618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.907890 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.907897 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.907990 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.911846 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.912704 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.912708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.912710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.912903 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.913350 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.913354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.913356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.913641 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.913902 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.914258 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.914507 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 15:56:29.918160 13376 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } attempt: 4
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610989, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
Sep 28 15:56:29 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610988, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
I20260928 15:56:29.928422 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.912437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53148)
0928 15:56:29.912717 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.912777 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:29.912866 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:29.913178 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:29.913184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:29.913567 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:29.913768 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.913780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.915236 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.915244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.916613 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.916629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.917455 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.917461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.917712 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.918317 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:29.918342 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:29.921380 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:29.925267 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.925282 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.925286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.925845 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.926282 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.926287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.926292 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.926457 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.927202 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:29.927210 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:29.927395 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 15:56:29.927604 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 15:56:29.927950 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1}
I20260928 15:56:29.928911 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.912407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.22.157.252:44971)
0928 15:56:29.912621 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.912630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.912835 (+ 205us) server_negotiation.cc:407] Connection header received
0928 15:56:29.912951 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.912959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.913043 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.913133 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.913922 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.915108 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.916818 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.917298 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.917446 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.921614 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.921620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.921625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.921670 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.925132 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.925947 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.925956 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.925971 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.926145 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.926591 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.926596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.926600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.927002 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.927149 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.927718 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.928206 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":178,"server-negotiator.queue_time_us":69}
I20260928 15:56:29.935248 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.916610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.22.157.253:35627)
0928 15:56:29.916789 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.916795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.916861 (+ 66us) server_negotiation.cc:407] Connection header received
0928 15:56:29.917023 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.917030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.917093 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.917214 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.918669 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.919847 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.921740 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.922698 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.922830 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.928144 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.928151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.928157 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.928202 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.932054 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.932864 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.932870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.932873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.933019 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.933561 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.933566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.933569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.933979 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.934284 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.934632 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.934889 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:56:29.935259 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.918722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53160)
0928 15:56:29.918886 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:29.918905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:29.919000 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:29.920119 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:29.920125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:29.920664 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:29.920936 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.920949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.922572 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.922582 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.924201 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:29.924222 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.925377 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.925385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:29.925611 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.926233 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:29.926250 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:29.929032 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:29.932427 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.932433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.932438 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.933037 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.933510 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:29.933518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:29.933522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:29.933680 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:29.934318 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:29.934322 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:29.934505 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 15:56:29.934623 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 15:56:29.934898 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 15:56:29.936287 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:29.919064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.22.157.252:44971)
0928 15:56:29.919781 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:29.919784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:29.919802 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:56:29.919840 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:29.919843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:29.919890 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:29.920094 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:29.921130 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.922421 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.924384 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:29.925170 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:29.925348 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:29.929289 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:29.929296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:29.929300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:29.929335 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:29.932272 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.933190 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.933196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.933200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.933371 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:29.933809 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:29.933812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:29.933814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:29.934158 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:29.935526 (+ 1368us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:29.935617 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 15:56:29.935828 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":406,"threads_started":1}
I20260928 15:56:29.937021 13524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a: Bootstrap starting.
I20260928 15:56:29.940593 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:29.940973 13524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:29.941655 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b: Bootstrap starting.
I20260928 15:56:29.942170 13524 log.cc:824] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:29.945796 13524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a: No bootstrap required, opened a new log
I20260928 15:56:29.945844 13376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:29.947289 13376 log.cc:824] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:29.944118 13351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d6ac2ceb368344cf977627952ec09b1b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34786
I20260928 15:56:29.950714 13376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b: No bootstrap required, opened a new log
I20260928 15:56:29.956525 13524 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } }
I20260928 15:56:29.957020 13524 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e81b1f54eb4a71abec5aac0321184a, State: Initialized, Role: FOLLOWER
I20260928 15:56:29.957553 13524 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } }
I20260928 15:56:29.958349 13376 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } }
I20260928 15:56:29.958741 13376 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ac2ceb368344cf977627952ec09b1b, State: Initialized, Role: FOLLOWER
I20260928 15:56:29.958578 13533 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:29.959093 13533 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:29.959244 13376 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } }
I20260928 15:56:29.959995 13524 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:29.960268 13534 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:29.960815 13534 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:29.961540 13376 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:29.961674 13351 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:56:29.961907 13351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:29.965586 13351 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } }
I20260928 15:56:29.967335 13351 leader_election.cc:288] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [CANDIDATE]: Term 1 election: Requested vote from peers f5fe5e7d599b4be9a5afec403cdad988 (127.22.157.253:35627), d6e81b1f54eb4a71abec5aac0321184a (127.22.157.252:44971)
I20260928 15:56:29.968250 13499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6ac2ceb368344cf977627952ec09b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d6e81b1f54eb4a71abec5aac0321184a"
I20260928 15:56:29.968788 13499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:29.973491 13499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ac2ceb368344cf977627952ec09b1b in term 1.
I20260928 15:56:29.974516 13326 leader_election.cc:302] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ac2ceb368344cf977627952ec09b1b, d6e81b1f54eb4a71abec5aac0321184a; no voters:
I20260928 15:56:29.975117 13534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:29.975845 13534 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [term 1 LEADER]: Becoming Leader. State: Replica: d6ac2ceb368344cf977627952ec09b1b, State: Running, Role: LEADER
I20260928 15:56:29.976490 13534 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } }
W20260928 15:56:29.982129 13548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:56:29.982267 13548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:56:29.996361 13551 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ac2ceb368344cf977627952ec09b1b. Latest consensus state: current_term: 1 leader_uuid: "d6ac2ceb368344cf977627952ec09b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:29.996912 13551 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:29.977304 13457 master_runner.cc:186] Error getting master registration for 127.22.157.254:43865: OK, instance_id { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" instance_seqno: 1790610988765961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:56:29.999465 13457 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 15:56:30.004141 13330 tablet.cc:2369] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:56:30.006120 13555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 15:56:30.012549 13555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:30.024435 13450 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } attempt: 5
Sep 28 15:56:30 dist-test-slave-1vw0 krb5kdc[13296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610989, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
I20260928 15:56:30.059644 13452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:30.025325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:44971 (local address 127.0.0.1:53176)
0928 15:56:30.025719 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:30.025745 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:30.025911 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:30.026628 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:30.026636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:30.027326 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:30.027591 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:30.027607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.029607 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.029616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:30.031573 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:30.031594 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.033862 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.033872 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:30.034402 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:30.035557 (+ 1155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:30.035617 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:30.041137 (+ 5520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:30.050440 (+ 9303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:30.050449 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:30.050456 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:30.051992 (+ 1536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:30.053676 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:30.053684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:30.053691 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:30.056765 (+ 3074us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:30.057869 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:30.057897 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:30.058173 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 15:56:30.058414 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 15:56:30.058967 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260928 15:56:30.061354 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:30.025375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.22.157.252:44971)
0928 15:56:30.025553 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.025561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.026049 (+ 488us) server_negotiation.cc:407] Connection header received
0928 15:56:30.026139 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.026144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.026264 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.026364 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.027842 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.029355 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.031873 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.032859 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.033015 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:30.041523 (+ 8508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:30.041534 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:30.041542 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:30.041601 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:30.049946 (+ 8345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:30.052442 (+ 2496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:30.052482 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:30.052488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:30.053200 (+ 712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:30.056902 (+ 3702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:30.056910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:30.056914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:30.057662 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:30.060036 (+ 2374us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:30.060303 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 15:56:30.060756 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 15:56:30.063778 13499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [term 1 FOLLOWER]: Refusing update from remote peer d6ac2ceb368344cf977627952ec09b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:56:30.065163 13534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:56:30.067909 13450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988: Bootstrap starting.
I20260928 15:56:30.072207 13450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:30.074347 13450 log.cc:824] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:30.084054 13450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988: No bootstrap required, opened a new log
W20260928 15:56:30.093874 13478 tablet.cc:2369] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:56:30.096856 13450 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } }
I20260928 15:56:30.097338 13450 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fe5e7d599b4be9a5afec403cdad988, State: Initialized, Role: FOLLOWER
I20260928 15:56:30.097901 13450 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } }
I20260928 15:56:30.079329 13309 master_runner.cc:186] Error getting master registration for 127.22.157.253:35627: OK, instance_id { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" instance_seqno: 1790610989294115 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:56:30.105039 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:30.105540 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:30.107692 13450 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:30.107758 13425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6ac2ceb368344cf977627952ec09b1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f5fe5e7d599b4be9a5afec403cdad988"
I20260928 15:56:30.108340 13425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:30.111964 13425 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ac2ceb368344cf977627952ec09b1b in term 1.
I20260928 15:56:30.113054 13424 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [term 1 FOLLOWER]: Refusing update from remote peer d6ac2ceb368344cf977627952ec09b1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:56:30.123683 13533 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ac2ceb368344cf977627952ec09b1b. Latest consensus state: current_term: 1 leader_uuid: "d6ac2ceb368344cf977627952ec09b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:30.124063 13533 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:30.114588 13534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:56:30.126593 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:30.043261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.22.157.254:43865)
0928 15:56:30.043474 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.043481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.057463 (+ 13982us) server_negotiation.cc:407] Connection header received
0928 15:56:30.057756 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.057764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.057856 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.057927 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.063481 (+ 5554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.064633 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.069111 (+ 4478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.069817 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.069932 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:30.096134 (+ 26202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:30.096142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:30.096147 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:30.096200 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:30.100687 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:30.116001 (+ 15314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:30.116011 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:30.116018 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:30.116236 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:30.116903 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:30.116911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:30.116917 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:30.117420 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:30.117805 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:30.125949 (+ 8144us) server_negotiation.cc:299] Negotiation successful
0928 15:56:30.126231 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:56:30.133343 13309 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:56:30.136683 13454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:30.025987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.22.157.253:35627)
0928 15:56:30.026278 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.026287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.044625 (+ 18338us) server_negotiation.cc:407] Connection header received
0928 15:56:30.045792 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.045798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.045885 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.045962 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.051743 (+ 5781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.052877 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.054789 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.055439 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.055567 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:30.104128 (+ 48561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:30.104138 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:30.104143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:30.104206 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:30.108412 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:30.118455 (+ 10043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:30.118465 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:30.118472 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:30.118708 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:30.124917 (+ 6209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:30.124927 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:30.124933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:30.125443 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:30.125741 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:30.135781 (+ 10040us) server_negotiation.cc:299] Negotiation successful
0928 15:56:30.136132 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260928 15:56:30.140911 13551 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6ac2ceb368344cf977627952ec09b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:30.141260 13551 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:30.144635 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:30.032096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.22.157.252:44971)
0928 15:56:30.033679 (+ 1583us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.033687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.048887 (+ 15200us) server_negotiation.cc:407] Connection header received
0928 15:56:30.049495 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.049500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.049582 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.049654 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.054857 (+ 5203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.056031 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.059132 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:30.059868 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:30.059973 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:30.119391 (+ 59418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:30.119399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:30.119406 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:30.119460 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:30.124073 (+ 4613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:30.135126 (+ 11053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:30.135135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:30.135140 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:30.135398 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:30.141981 (+ 6583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:30.141995 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:30.142000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:30.142547 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:30.142888 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:30.143722 (+ 834us) server_negotiation.cc:299] Negotiation successful
0928 15:56:30.144084 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114}
I20260928 15:56:30.146975 13570 mvcc.cc:205] Tried to move back new op lower bound from 7334342615251255296 to 7334342614949359616. Current Snapshot: MvccSnapshot[applied={T|T < 7334342615251255296}]
I20260928 15:56:30.148749 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ac2ceb368344cf977627952ec09b1b. Latest consensus state: current_term: 1 leader_uuid: "d6ac2ceb368344cf977627952ec09b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:30.149114 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:30.151703 13555 catalog_manager.cc:1347] Generated new cluster ID: fd2fe20e3ffc4dc4b5710201e54352da
I20260928 15:56:30.151824 13555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:30.152915 13533 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6ac2ceb368344cf977627952ec09b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:30.153216 13533 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:30.163139 13583 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6ac2ceb368344cf977627952ec09b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:30.163614 13583 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:30.181221 13555 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:30.183284 13555 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:30.201097 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6ac2ceb368344cf977627952ec09b1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6ac2ceb368344cf977627952ec09b1b" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 43865 } } peers { permanent_uuid: "f5fe5e7d599b4be9a5afec403cdad988" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 35627 } } peers { permanent_uuid: "d6e81b1f54eb4a71abec5aac0321184a" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 44971 } } }
I20260928 15:56:30.201486 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:30.210219 13555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6ac2ceb368344cf977627952ec09b1b: Generated new TSK 0
I20260928 15:56:30.211402 13555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:30.235596 13591 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 15:56:30.297699 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.292827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.22.157.253:35627)
0928 15:56:30.293072 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.293081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.295152 (+ 2071us) server_negotiation.cc:407] Connection header received
0928 15:56:30.295515 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.295521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.295632 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.295748 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.297347 (+ 1599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33820: BlockingRecv error: recv got EOF from 127.0.0.1:33820 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 15:56:30.301194 13528 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.292363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34796 (local address 127.22.157.254:43865)
0928 15:56:30.292608 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.292615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.298940 (+ 6325us) server_negotiation.cc:407] Connection header received
0928 15:56:30.299385 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.299394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.299505 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.299594 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.300876 (+ 1282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34796: BlockingRecv error: recv got EOF from 127.0.0.1:34796 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 15:56:30.304165 13527 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.295570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.22.157.252:44971)
0928 15:56:30.295751 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.295757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.302076 (+ 6319us) server_negotiation.cc:407] Connection header received
0928 15:56:30.302350 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.302359 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.302466 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.302554 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.303798 (+ 1244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53206: BlockingRecv error: recv got EOF from 127.0.0.1:53206 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 15:56:30.336977 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.334385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.22.157.253:35627)
0928 15:56:30.334602 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.334609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.335165 (+ 556us) server_negotiation.cc:407] Connection header received
0928 15:56:30.335411 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.335418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.335562 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.335688 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.336630 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33830: BlockingRecv error: recv got EOF from 127.0.0.1:33830 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 15:56:30.347563 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.342119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.22.157.253:35627)
0928 15:56:30.342315 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.342321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.345965 (+ 3644us) server_negotiation.cc:407] Connection header received
0928 15:56:30.346180 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.346186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.346286 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.346409 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.347340 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33836: BlockingRecv error: recv got EOF from 127.0.0.1:33836 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 15:56:30.358848 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.352697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.22.157.253:35627)
0928 15:56:30.352899 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.352905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.357339 (+ 4434us) server_negotiation.cc:407] Connection header received
0928 15:56:30.357534 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.357541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.357641 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.357755 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.358601 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33842: BlockingRecv error: recv got EOF from 127.0.0.1:33842 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 15:56:30.368494 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.366113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.22.157.253:35627)
0928 15:56:30.366345 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.366351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.366883 (+ 532us) server_negotiation.cc:407] Connection header received
0928 15:56:30.367083 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.367088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.367197 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.367314 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.368219 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33852: BlockingRecv error: recv got EOF from 127.0.0.1:33852 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260928 15:56:30.384688 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.382399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.22.157.253:35627)
0928 15:56:30.382610 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.382617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.383111 (+ 494us) server_negotiation.cc:407] Connection header received
0928 15:56:30.383317 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.383324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.383438 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.383559 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.384422 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33862: BlockingRecv error: recv got EOF from 127.0.0.1:33862 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 15:56:30.412240 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.409804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.22.157.253:35627)
0928 15:56:30.410000 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.410006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.410609 (+ 603us) server_negotiation.cc:407] Connection header received
0928 15:56:30.410811 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.410819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.410927 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.411058 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.411994 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33870: BlockingRecv error: recv got EOF from 127.0.0.1:33870 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 15:56:30.453214 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.450757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.22.157.253:35627)
0928 15:56:30.450939 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.450945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.451485 (+ 540us) server_negotiation.cc:407] Connection header received
0928 15:56:30.451682 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.451689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.451812 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.451958 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.452890 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33882: BlockingRecv error: recv got EOF from 127.0.0.1:33882 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 15:56:30.524981 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.522484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.22.157.253:35627)
0928 15:56:30.522722 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.522728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.523298 (+ 570us) server_negotiation.cc:407] Connection header received
0928 15:56:30.523520 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.523527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.523633 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.523758 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.524696 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33896: BlockingRecv error: recv got EOF from 127.0.0.1:33896 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 15:56:30.659978 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.657938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.22.157.253:35627)
0928 15:56:30.658221 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.658229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.658504 (+ 275us) server_negotiation.cc:407] Connection header received
0928 15:56:30.658761 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.658766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.658850 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.658966 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.659685 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33910: BlockingRecv error: recv got EOF from 127.0.0.1:33910 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260928 15:56:30.921818 13454 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:30.919709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.22.157.253:35627)
0928 15:56:30.919898 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:30.919902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:30.920268 (+ 366us) server_negotiation.cc:407] Connection header received
0928 15:56:30.920424 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:30.920428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:30.920501 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:30.920633 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:30.921543 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33914: BlockingRecv error: recv got EOF from 127.0.0.1:33914 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 15:56:30.984212 13548 catalog_manager.cc:1259] Loaded cluster ID: fd2fe20e3ffc4dc4b5710201e54352da
I20260928 15:56:30.984316 13548 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a: loading cluster ID for follower catalog manager: success
I20260928 15:56:30.986801 13548 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a: acquiring CA information for follower catalog manager: success
I20260928 15:56:30.988525 13548 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6e81b1f54eb4a71abec5aac0321184a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:56:31.240007 13591 catalog_manager.cc:1259] Loaded cluster ID: fd2fe20e3ffc4dc4b5710201e54352da
I20260928 15:56:31.240126 13591 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988: loading cluster ID for follower catalog manager: success
I20260928 15:56:31.242908 13591 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988: acquiring CA information for follower catalog manager: success
I20260928 15:56:31.245174 13591 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5fe5e7d599b4be9a5afec403cdad988: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:56:31.446971 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:31.438412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.22.157.253:35627)
0928 15:56:31.439109 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:31.439116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:31.439343 (+ 227us) server_negotiation.cc:407] Connection header received
0928 15:56:31.439583 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:31.439592 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:31.439839 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:31.440075 (+ 236us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:31.440772 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.442107 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.443739 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.444495 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.444649 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:31.445350 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:31.446173 (+ 823us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:31.446344 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:31.446426 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 15:56:31.446542 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":323,"threads_started":1}
I20260928 15:56:31.470330 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:31.462096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.22.157.252:44971)
0928 15:56:31.463261 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:31.463266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:31.463286 (+ 20us) server_negotiation.cc:407] Connection header received
0928 15:56:31.463407 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:31.463410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:31.463569 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:31.463716 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:31.464247 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.465082 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.467368 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.467884 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.468039 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:31.468875 (+ 836us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:31.469532 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:31.469658 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:31.469868 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 15:56:31.469949 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":925,"threads_started":1}
I20260928 15:56:31.498009 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:31.477946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.22.157.253:35627)
0928 15:56:31.478133 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:31.478139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:31.490230 (+ 12091us) server_negotiation.cc:407] Connection header received
0928 15:56:31.490694 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:31.490700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:31.490796 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:31.490869 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:31.491758 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.492937 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.495457 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.496140 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.496265 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:31.497144 (+ 879us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:31.497292 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:31.497359 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:31.497584 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 15:56:31.497686 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 15:56:31.498024 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:31.478054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.22.157.252:44971)
0928 15:56:31.478271 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:31.478280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:31.489675 (+ 11395us) server_negotiation.cc:407] Connection header received
0928 15:56:31.490296 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:31.490305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:31.490437 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:31.490516 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:31.491536 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.492884 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.495456 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.496083 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.496214 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:31.497209 (+ 995us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:31.497356 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:31.497429 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:31.497666 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 15:56:31.497751 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 15:56:31.503235 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:31.477511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.22.157.254:43865)
0928 15:56:31.495241 (+ 17730us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:31.495248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:31.495270 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:56:31.495391 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:31.495396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:31.495656 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:31.495860 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:31.497076 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.498159 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.500753 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:31.501445 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:31.501595 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:31.502108 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:31.502597 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:31.502711 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:31.502829 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 15:56:31.502919 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17577,"thread_start_us":18469,"threads_started":1}
I20260928 15:56:31.509698 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13309
I20260928 15:56:31.510845 13371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:31.625252 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 13309
I20260928 15:56:31.648571 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13383
I20260928 15:56:31.649852 13445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:31.760144 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 13383
I20260928 15:56:31.782943 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13457
I20260928 15:56:31.784462 13519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:31.896586 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 13457
2026-09-28T15:56:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5066 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 15:56:32 dist-test-slave-1vw0 krb5kdc[13678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 15:56:32 dist-test-slave-1vw0 krb5kdc[13678](info): set up 2 sockets
Sep 28 15:56:32 dist-test-slave-1vw0 krb5kdc[13678](info): commencing operation
krb5kdc: starting...
W20260928 15:56:33.049779 23159 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/test-user.keytab.
Sep 28 15:56:33 dist-test-slave-1vw0 krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T15:56:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T15:56:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
Entry for principal kudu/127.22.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab.
I20260928 15:56:33.358019 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:38345
--webserver_interface=127.22.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:38345,127.22.157.253:37551,127.22.157.252:38537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 15:56:33.721833 13691 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:56:33.722198 13691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:33.722278 13691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:33.722366 13691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:33.730052 13691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:33.730170 13691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:33.730243 13691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:33.730302 13691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:33.730376 13691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:33.741110 13691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:38345,127.22.157.253:37551,127.22.157.252:38537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.254.keytab
--principal=kudu/127.22.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.254:38345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:33.742447 13691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:33.743884 13691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:33.756595 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:33.756511 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:33.756565 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:33.757506 13691 server_base.cc:1034] running on GCE node
I20260928 15:56:33.758484 13691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:33.760059 13691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:33.761289 13691 hybrid_clock.cc:648] HybridClock initialized: now 1790610993761224 us; error 32 us; skew 500 ppm
I20260928 15:56:33.761662 13691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:33 dist-test-slave-1vw0 krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:33.771404 13691 init.cc:377] Logged in from keytab as kudu/127.22.157.254@KRBTEST.COM (short username kudu)
I20260928 15:56:33.773212 13691 webserver.cc:460] Webserver started at http://127.22.157.254:41695/ using document root <none> and password file <none>
I20260928 15:56:33.773972 13691 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:33.774108 13691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:33.774468 13691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:33.777679 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/data/instance:
uuid: "2e3c82b5a0d04f50987723e58ce1684c"
format_stamp: "Formatted at 2026-09-28 15:56:33 on dist-test-slave-1vw0"
server_key: "054e02dcd29b48368c94f36e140c9ca9"
server_key_iv: "06079cc1713dfafdeafe35c412856b98"
server_key_version: "encryptionkey@0"
I20260928 15:56:33.778533 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/wal/instance:
uuid: "2e3c82b5a0d04f50987723e58ce1684c"
format_stamp: "Formatted at 2026-09-28 15:56:33 on dist-test-slave-1vw0"
server_key: "054e02dcd29b48368c94f36e140c9ca9"
server_key_iv: "06079cc1713dfafdeafe35c412856b98"
server_key_version: "encryptionkey@0"
I20260928 15:56:33.789744 13691 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.005s
I20260928 15:56:33.794659 13706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:33.796185 13691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 15:56:33.796414 13691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/wal
uuid: "2e3c82b5a0d04f50987723e58ce1684c"
format_stamp: "Formatted at 2026-09-28 15:56:33 on dist-test-slave-1vw0"
server_key: "054e02dcd29b48368c94f36e140c9ca9"
server_key_iv: "06079cc1713dfafdeafe35c412856b98"
server_key_version: "encryptionkey@0"
I20260928 15:56:33.796670 13691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:33.810915 13691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:33.815045 13691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:33.815332 13691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:33.834532 13691 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.254:38345
I20260928 15:56:33.834554 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.254:38345 every 8 connection(s)
I20260928 15:56:33.835983 13691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/data/info.pb
I20260928 15:56:33.839310 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 13691
I20260928 15:56:33.839505 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-0/wal/instance
I20260928 15:56:33.839938 23159 external_mini_cluster.cc:1428] Setting key 2f6428f6f8b1621ca6bed9443e26b683
I20260928 15:56:33.840184 13758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:33.847427 13758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:33 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.253@KRBTEST.COM; defaulting to no policy
I20260928 15:56:33.873196 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:33.852272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.22.157.254:38345)
0928 15:56:33.853064 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:33.853075 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:33.853126 (+ 51us) server_negotiation.cc:407] Connection header received
0928 15:56:33.854463 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:33.854473 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:33.854802 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:33.855038 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:33.856213 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:33.858224 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:33.859790 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:33.860874 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:33.861071 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:33.864541 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:33.864555 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:33.864576 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:33.864627 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:33.868816 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:33.869528 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:33.869537 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:33.869541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:33.869754 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:33.870192 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:33.870198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:33.870201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:33.870966 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:33.871122 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:33.871708 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 15:56:33.871999 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":557,"thread_start_us":292,"threads_started":1}
I20260928 15:56:33.873250 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:33.850960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:38345 (local address 127.0.0.1:45390)
0928 15:56:33.851804 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:33.851851 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:33.854043 (+ 2192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:33.855046 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:33.855052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:33.855633 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:33.855973 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:33.855990 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:33.858418 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:33.858422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:33.859632 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:33.859646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:33.861121 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:33.861127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:33.861273 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:33.861999 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:33.862023 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:33.864290 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:33.868964 (+ 4674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:33.868974 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:33.868985 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:33.869386 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:33.869880 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:33.869883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:33.869885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:33.870066 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:33.871111 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:33.871118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:33.871295 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 15:56:33.871530 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 15:56:33.872142 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":97,"thread_start_us":355,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.253@KRBTEST.COM" created.
I20260928 15:56:33.876286 13758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:56:33.877506 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:33.877133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47634)
0928 15:56:33.877297 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:33.877393 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 15:56:33.877795 13708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
I20260928 15:56:33.879518 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:33.879065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47646)
0928 15:56:33.879248 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:33.879344 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 15:56:33.879755 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:37551: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab.
Entry for principal kudu/127.22.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab.
I20260928 15:56:33.907701 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.253:37551
--webserver_interface=127.22.157.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab
--principal=kudu/127.22.157.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:38345,127.22.157.253:37551,127.22.157.252:38537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:56:33.929147 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } attempt: 1
I20260928 15:56:33.930507 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:33.930010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47658)
0928 15:56:33.930246 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:33.930372 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 15:56:33.931622 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:33.931245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47666)
0928 15:56:33.931399 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:33.931465 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 15:56:33.931804 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:37551: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
I20260928 15:56:34.000079 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } attempt: 2
I20260928 15:56:34.001351 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.000902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47670)
0928 15:56:34.001090 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.001215 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 15:56:34.002442 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.002089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47686)
0928 15:56:34.002260 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.002321 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:56:34.002661 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:37551: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
I20260928 15:56:34.093947 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } attempt: 3
I20260928 15:56:34.095122 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.094687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47698)
0928 15:56:34.094892 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.094989 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 15:56:34.096303 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.095836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47702)
0928 15:56:34.096014 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.096107 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 15:56:34.096539 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:37551: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
I20260928 15:56:34.239862 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } attempt: 4
I20260928 15:56:34.241279 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.240578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47704)
0928 15:56:34.241004 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.241113 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
I20260928 15:56:34.242410 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.241998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:47714)
0928 15:56:34.242161 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.242260 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 15:56:34.242607 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.253:37551: Network error: Client connection negotiation failed: client connection to 127.22.157.253:37551: connect: Connection refused (error 111)
W20260928 15:56:34.251246 13765 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:56:34.251590 13765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:34.251662 13765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:34.251742 13765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 15:56:34.259258 13765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:34.259356 13765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:34.259413 13765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:34.259465 13765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:34.259527 13765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:34.269877 13765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:38345,127.22.157.253:37551,127.22.157.252:38537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.253.keytab
--principal=kudu/127.22.157.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.253:37551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.157.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:34.271201 13765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:34.272501 13765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 15:56:34.284417 13773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:34.284476 13771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:34.284780 13770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:34.285785 13765 server_base.cc:1034] running on GCE node
I20260928 15:56:34.286615 13765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:34.287972 13765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:34.289211 13765 hybrid_clock.cc:648] HybridClock initialized: now 1790610994289144 us; error 30 us; skew 500 ppm
I20260928 15:56:34.289554 13765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610994, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:34.292994 13765 init.cc:377] Logged in from keytab as kudu/127.22.157.253@KRBTEST.COM (short username kudu)
I20260928 15:56:34.294749 13765 webserver.cc:460] Webserver started at http://127.22.157.253:43585/ using document root <none> and password file <none>
I20260928 15:56:34.295435 13765 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:34.295522 13765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:34.295779 13765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:34.298769 13765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/data/instance:
uuid: "c40f13097b644d8aaca795b95ae73a0d"
format_stamp: "Formatted at 2026-09-28 15:56:34 on dist-test-slave-1vw0"
server_key: "8900a5325766204a3642a9f81588107f"
server_key_iv: "2b874fd4a84e32c839b28520525e3b75"
server_key_version: "encryptionkey@0"
I20260928 15:56:34.299515 13765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/wal/instance:
uuid: "c40f13097b644d8aaca795b95ae73a0d"
format_stamp: "Formatted at 2026-09-28 15:56:34 on dist-test-slave-1vw0"
server_key: "8900a5325766204a3642a9f81588107f"
server_key_iv: "2b874fd4a84e32c839b28520525e3b75"
server_key_version: "encryptionkey@0"
I20260928 15:56:34.304889 13765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 15:56:34.308285 13780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:34.309742 13765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:56:34.309887 13765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/wal
uuid: "c40f13097b644d8aaca795b95ae73a0d"
format_stamp: "Formatted at 2026-09-28 15:56:34 on dist-test-slave-1vw0"
server_key: "8900a5325766204a3642a9f81588107f"
server_key_iv: "2b874fd4a84e32c839b28520525e3b75"
server_key_version: "encryptionkey@0"
I20260928 15:56:34.310070 13765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:34.330943 13765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:34.335577 13765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:34.335813 13765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:34.354841 13765 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.253:37551
I20260928 15:56:34.354861 13831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.253:37551 every 8 connection(s)
I20260928 15:56:34.356240 13765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/data/info.pb
I20260928 15:56:34.357008 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 13765
I20260928 15:56:34.357200 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-1/wal/instance
I20260928 15:56:34.357764 23159 external_mini_cluster.cc:1428] Setting key a32a8f187d4c0a601c6883d23fa23a55
I20260928 15:56:34.360155 13832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:34.366891 13832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610994, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.157.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.157.252@KRBTEST.COM" created.
I20260928 15:56:34.388149 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.369794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46886 (local address 127.22.157.254:38345)
0928 15:56:34.369980 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.369985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.371346 (+ 1361us) server_negotiation.cc:407] Connection header received
0928 15:56:34.373390 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.373395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.373487 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.373609 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.375244 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.375949 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.377608 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.378141 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.378279 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.381876 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.381881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.381884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.381920 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.384884 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.385694 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.385704 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.385706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.385858 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.386357 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.386364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.386372 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.386676 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.386836 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.387572 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.387816 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:56:34.389263 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.370366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:38345 (local address 127.0.0.1:46886)
0928 15:56:34.371169 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.371213 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:34.373113 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:34.373846 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:34.373854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:34.374478 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:34.375055 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.375073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.376129 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.376133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.377429 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.377441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.378390 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.378398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.378648 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.379364 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:34.379389 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:34.381632 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:34.385037 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.385047 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.385066 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.385547 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.385977 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.385983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.385986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.386238 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.386816 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:34.386824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:34.387197 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 15:56:34.387463 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 15:56:34.388005 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":348,"threads_started":1}
I20260928 15:56:34.391131 13832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610994, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
I20260928 15:56:34.408124 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.392236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.22.157.253:37551)
0928 15:56:34.392942 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.392953 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.392985 (+ 32us) server_negotiation.cc:407] Connection header received
0928 15:56:34.393039 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.393049 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.393298 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.393455 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.394329 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.396048 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.397721 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.398583 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.398745 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.402339 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.402351 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.402365 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.402395 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.405390 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.406041 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.406052 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.406060 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.406176 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.406549 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.406556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.406569 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.406925 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.407175 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.407516 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.407745 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":332,"threads_started":1}
I20260928 15:56:34.408092 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.391991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:55242)
0928 15:56:34.392199 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.392220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:34.392328 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:34.393468 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:34.393475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:34.393942 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:34.394136 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.394149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.396191 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.396198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.397572 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.397587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.398764 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.398770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.398945 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.399574 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:34.399592 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:34.402110 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:34.405537 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.405543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.405547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.405906 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.406294 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.406300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.406303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.406446 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.407074 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:34.407079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:34.407222 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 15:56:34.407368 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 15:56:34.407651 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 15:56:34.411090 13832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:56:34.412520 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.411895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33482)
0928 15:56:34.412127 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.412285 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 15:56:34.412899 13782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
I20260928 15:56:34.414762 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.414297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33498)
0928 15:56:34.414479 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.414587 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 15:56:34.415001 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:38537: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab.
Entry for principal kudu/127.22.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab.
I20260928 15:56:34.424458 23159 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1s6B80/build/asan/bin/kudu
/tmp/dist-test-task1s6B80/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.252:38537
--webserver_interface=127.22.157.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab
--principal=kudu/127.22.157.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.157.212:45447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.157.254:38345,127.22.157.253:37551,127.22.157.252:38537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 15:56:34.464403 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } attempt: 1
I20260928 15:56:34.465695 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.465201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33502)
0928 15:56:34.465410 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.465535 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 15:56:34.466956 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.466538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33514)
0928 15:56:34.466735 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.466823 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 15:56:34.467361 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:38537: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
I20260928 15:56:34.535605 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } attempt: 2
I20260928 15:56:34.536859 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.536326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33528)
0928 15:56:34.536529 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.536638 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 15:56:34.537892 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.537516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33542)
0928 15:56:34.537720 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.537785 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260928 15:56:34.538121 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:38537: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
I20260928 15:56:34.541886 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } attempt: 5
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
I20260928 15:56:34.556540 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.542686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:55258)
0928 15:56:34.542913 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.542934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:34.543067 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:34.543586 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:34.543590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:34.543969 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:34.544153 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.544162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.545610 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.545614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.546480 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.546494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.547614 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.547619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.547785 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.548545 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:34.548564 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:34.550969 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:34.554075 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.554080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.554084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.554479 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.554864 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.554869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.554873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.555024 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.555563 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:34.555568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:34.555726 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 15:56:34.555875 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 15:56:34.556165 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 15:56:34.556529 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.542877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.22.157.253:37551)
0928 15:56:34.543104 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.543112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.543138 (+ 26us) server_negotiation.cc:407] Connection header received
0928 15:56:34.543213 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.543220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.543339 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.543494 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.544322 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.545445 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.546643 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.547311 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.547441 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.551192 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.551197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.551200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.551235 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.553907 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.554609 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.554616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.554619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.554746 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.555135 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.555141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.555144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.555437 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.555596 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.555971 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.556194 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 15:56:34.558130 13758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
I20260928 15:56:34.559213 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.558800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33554)
0928 15:56:34.558998 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.559074 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 15:56:34.560199 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.559841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33566)
0928 15:56:34.560026 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.560092 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 15:56:34.560426 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:38537: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
I20260928 15:56:34.611663 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } attempt: 1
I20260928 15:56:34.613081 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.612475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33578)
0928 15:56:34.612752 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.612870 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 15:56:34.614264 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.613853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33580)
0928 15:56:34.614015 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.614106 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 15:56:34.614711 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:38537: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
I20260928 15:56:34.629340 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } attempt: 3
I20260928 15:56:34.630542 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.630057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33584)
0928 15:56:34.630256 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.630361 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 15:56:34.631903 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.631362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33586)
0928 15:56:34.631694 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.631780 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
W20260928 15:56:34.632148 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:38537: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
I20260928 15:56:34.682981 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } attempt: 2
I20260928 15:56:34.684352 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.683701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33588)
0928 15:56:34.683980 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.684112 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 15:56:34.685612 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.685173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33594)
0928 15:56:34.685344 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.685442 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 15:56:34.685894 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:38537: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
W20260928 15:56:34.770570 13839 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 15:56:34.770905 13839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 15:56:34.770967 13839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 15:56:34.771049 13839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 15:56:34.775379 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } attempt: 4
I20260928 15:56:34.776530 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.776026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33606)
0928 15:56:34.776272 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.776366 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 15:56:34.777853 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.777519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33612)
0928 15:56:34.777681 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.777752 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 15:56:34.778139 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:38537: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
W20260928 15:56:34.779265 13839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 15:56:34.779381 13839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 15:56:34.779492 13839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 15:56:34.779595 13839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 15:56:34.779686 13839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 15:56:34.790211 13839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.157.212:45447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.157.254:38345,127.22.157.253:37551,127.22.157.252:38537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/krb5kdc/kudu_127.22.157.252.keytab
--principal=kudu/127.22.157.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.157.252:38537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.157.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 14:11:15 UTC on ccycloud.quasar-ubnjdo.root.comops.site
build id 4677204
ASAN enabled
I20260928 15:56:34.791531 13839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 15:56:34.792141 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } attempt: 3
I20260928 15:56:34.793221 13839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 15:56:34.793514 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.792889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33620)
0928 15:56:34.793158 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.793281 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 15:56:34.794657 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.794233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33636)
0928 15:56:34.794399 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.794511 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 15:56:34.794920 13758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.157.252:38537: Network error: Client connection negotiation failed: client connection to 127.22.157.252:38537: connect: Connection refused (error 111)
W20260928 15:56:34.806890 13844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:34.806963 13847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 15:56:34.806954 13845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 15:56:34.813122 13839 server_base.cc:1034] running on GCE node
I20260928 15:56:34.814018 13839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 15:56:34.815505 13839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 15:56:34.816813 13839 hybrid_clock.cc:648] HybridClock initialized: now 1790610994816678 us; error 61 us; skew 500 ppm
I20260928 15:56:34.817166 13839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610994, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 15:56:34.820361 13839 init.cc:377] Logged in from keytab as kudu/127.22.157.252@KRBTEST.COM (short username kudu)
I20260928 15:56:34.821906 13839 webserver.cc:460] Webserver started at http://127.22.157.252:46103/ using document root <none> and password file <none>
I20260928 15:56:34.822587 13839 fs_manager.cc:359] Metadata directory not provided
I20260928 15:56:34.822674 13839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 15:56:34.822938 13839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 15:56:34.825878 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/data/instance:
uuid: "18582b2d8b4d4e23baa03d7132eb4bd2"
format_stamp: "Formatted at 2026-09-28 15:56:34 on dist-test-slave-1vw0"
server_key: "26be804529bef853fbf1e532594d3059"
server_key_iv: "9648a8f847eadf80e1c59d2ddbbcef73"
server_key_version: "encryptionkey@0"
I20260928 15:56:34.826709 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/wal/instance:
uuid: "18582b2d8b4d4e23baa03d7132eb4bd2"
format_stamp: "Formatted at 2026-09-28 15:56:34 on dist-test-slave-1vw0"
server_key: "26be804529bef853fbf1e532594d3059"
server_key_iv: "9648a8f847eadf80e1c59d2ddbbcef73"
server_key_version: "encryptionkey@0"
I20260928 15:56:34.831797 13839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 15:56:34.835453 13854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 15:56:34.837021 13839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 15:56:34.837201 13839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/data,/tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/wal
uuid: "18582b2d8b4d4e23baa03d7132eb4bd2"
format_stamp: "Formatted at 2026-09-28 15:56:34 on dist-test-slave-1vw0"
server_key: "26be804529bef853fbf1e532594d3059"
server_key_iv: "9648a8f847eadf80e1c59d2ddbbcef73"
server_key_version: "encryptionkey@0"
I20260928 15:56:34.837421 13839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 15:56:34.869716 13839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 15:56:34.873965 13839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 15:56:34.874221 13839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 15:56:34.892879 13839 rpc_server.cc:307] RPC server started. Bound to: 127.22.157.252:38537
I20260928 15:56:34.892899 13905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.157.252:38537 every 8 connection(s)
I20260928 15:56:34.894261 13839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/data/info.pb
I20260928 15:56:34.897580 23159 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1s6B80/build/asan/bin/kudu as pid 13839
I20260928 15:56:34.897786 23159 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1s6B80/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790610618730370-23159-0/minicluster-data/master-2/wal/instance
I20260928 15:56:34.898322 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 15:56:34.898337 23159 external_mini_cluster.cc:1428] Setting key 0c94aa6f0394d279d1dbcf1873671a73
I20260928 15:56:34.908015 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610994, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
I20260928 15:56:34.922494 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.900207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.22.157.252:38537)
0928 15:56:34.901088 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.901100 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.901150 (+ 50us) server_negotiation.cc:407] Connection header received
0928 15:56:34.903110 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.903132 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.903541 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.903864 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.905660 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.906923 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.908717 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.909500 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.909623 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.914327 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.914334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.914343 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.914379 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.917976 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.918657 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.918673 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.918682 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.918875 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.919249 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.919255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.919264 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.920013 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.920229 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.920865 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.921328 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":371,"threads_started":1}
I20260928 15:56:34.927762 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.911025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.254:38345 (local address 127.0.0.1:46892)
0928 15:56:34.912008 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.912047 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:34.912211 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:34.912656 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:34.912665 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:34.913446 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:34.913692 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.913700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.915015 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.915019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.916906 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.916923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.917683 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.917692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.917900 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.918567 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:34.918615 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:34.921436 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:34.924983 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.924998 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.925006 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.925544 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.926035 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.926039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.926042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.926251 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.926889 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:34.926898 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:34.927097 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 15:56:34.927247 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 15:56:34.927478 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":615,"threads_started":1}
I20260928 15:56:34.928089 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.911006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.22.157.254:38345)
0928 15:56:34.911639 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.911647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.912151 (+ 504us) server_negotiation.cc:407] Connection header received
0928 15:56:34.912346 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.912355 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.912451 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.912565 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.913844 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.914854 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.917059 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.917485 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.917582 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.921700 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.921706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.921709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.921745 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.924835 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.925678 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.925684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.925687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.925887 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.926396 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.926399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.926402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.926739 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.926910 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.927430 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.927689 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":347,"threads_started":1}
I20260928 15:56:34.929484 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.254@KRBTEST.COM
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610994, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
I20260928 15:56:34.942678 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.925889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.22.157.254:38345)
0928 15:56:34.926509 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.926513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.926531 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:56:34.926571 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.926574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.926633 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.926716 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.928537 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.929744 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.931523 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.931958 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.932090 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.936176 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.936182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.936185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.936218 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.939749 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.940392 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.940397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.940401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.940544 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.940974 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.940977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.940979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.941341 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.941601 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.941998 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.942246 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":273,"threads_started":1}
I20260928 15:56:34.943238 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.930128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.253:37551 (local address 127.0.0.1:55264)
0928 15:56:34.930295 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.930314 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:34.930402 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:34.930686 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:34.930691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:34.931139 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:34.931313 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.931323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.932279 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.932287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.933355 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.933367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.934089 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.934095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.934299 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.934826 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:34.934842 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:34.936977 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:34.940550 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.940554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.940557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.941099 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.941488 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.941492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.941496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.941691 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.942298 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:34.942302 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:34.942492 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 15:56:34.942638 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 15:56:34.942923 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 15:56:34.943883 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.930277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55264 (local address 127.22.157.253:37551)
0928 15:56:34.930425 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.930429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.930447 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:56:34.930495 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.930499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.930555 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.930634 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.931464 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.932137 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.933503 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.933934 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.934059 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.937176 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.937183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.937188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.937232 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.940427 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.941210 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.941215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.941219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.941377 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.941813 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.941817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.941819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.942158 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.942311 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.943222 (+ 911us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.943509 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 15:56:34.944773 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610994, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.252@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.157.253@KRBTEST.COM
I20260928 15:56:34.959520 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.945461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33652)
0928 15:56:34.945861 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.945879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:34.945990 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:34.946269 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:34.946275 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:34.946734 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:34.946874 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.946882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.948096 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.948102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.949452 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.949470 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.950363 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.950369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.950630 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.951291 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:34.951323 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:34.953813 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:34.957005 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.957013 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.957018 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.957500 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.957841 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.957847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.957852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.958010 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.958506 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:34.958511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:34.958669 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 15:56:34.958808 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 15:56:34.959114 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260928 15:56:34.959575 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.945680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.22.157.252:38537)
0928 15:56:34.945803 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.945807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.945946 (+ 139us) server_negotiation.cc:407] Connection header received
0928 15:56:34.946087 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.946090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.946140 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.946212 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.947016 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.947985 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.949582 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.950204 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.950316 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.953983 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.953988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.953991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.954018 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.956876 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.957628 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.957631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.957633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.957734 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.958117 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.958120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.958122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.958383 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.958531 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.958892 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.959268 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":37}
I20260928 15:56:34.961180 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.944580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.22.157.253:37551)
0928 15:56:34.944762 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.944769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.944846 (+ 77us) server_negotiation.cc:407] Connection header received
0928 15:56:34.944985 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.944990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.945059 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.945155 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.946717 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.947776 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.949502 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.950185 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.950316 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.955256 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.955261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.955265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.955295 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.958300 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.958942 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.958947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.958953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.959084 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.959480 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.959484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.959487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.960017 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.960313 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.960599 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.960851 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 15:56:34.965451 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2: Bootstrap starting.
I20260928 15:56:34.965885 23159 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 15:56:34.969143 13906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:34.970510 13906 log.cc:824] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:34.968176 13733 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2e3c82b5a0d04f50987723e58ce1684c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46900
I20260928 15:56:34.972141 13758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } attempt: 4
I20260928 15:56:34.973862 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2: No bootstrap required, opened a new log
Sep 28 15:56:34 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610993, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.254@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
I20260928 15:56:34.985873 13906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } }
I20260928 15:56:34.986274 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.972825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.22.157.252:38537)
0928 15:56:34.973041 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:34.973047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:34.973068 (+ 21us) server_negotiation.cc:407] Connection header received
0928 15:56:34.973167 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:34.973172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:34.973238 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:34.973342 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:34.974224 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.975072 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.976328 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.976912 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.977051 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.980358 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:34.980363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:34.980367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:34.980400 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:34.983504 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.984198 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.984203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.984207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.984340 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:34.984777 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:34.984784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:34.984788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:34.985097 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:34.985284 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:34.985654 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 15:56:34.985865 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 15:56:34.986274 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:34.972723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33656)
0928 15:56:34.972907 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:34.972929 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:34.973048 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:34.973390 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:34.973394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:34.973837 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:34.974065 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.974076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.975228 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.975235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.976177 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:34.976190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:34.977061 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:34.977067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:34.977287 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:34.977890 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:34.977908 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:34.980166 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:34.983658 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.983666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.983670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.984074 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.984475 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:34.984480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:34.984482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:34.984622 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:34.985238 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:34.985244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:34.985408 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 15:56:34.985566 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 15:56:34.985863 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 15:56:34.986369 13906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18582b2d8b4d4e23baa03d7132eb4bd2, State: Initialized, Role: FOLLOWER
I20260928 15:56:34.986941 13906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } }
I20260928 15:56:34.988204 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:34.988902 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:34.989020 13906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:34.992342 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c: Bootstrap starting.
I20260928 15:56:34.996690 13758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:34.998093 13758 log.cc:824] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:35.002846 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c: No bootstrap required, opened a new log
W20260928 15:56:35.006309 13926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 15:56:35.006496 13926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 15:56:35.011262 13758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } }
I20260928 15:56:35.011746 13758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3c82b5a0d04f50987723e58ce1684c, State: Initialized, Role: FOLLOWER
I20260928 15:56:35.012324 13758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } }
I20260928 15:56:35.013715 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:35.014621 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:35.003568 13839 master_runner.cc:186] Error getting master registration for 127.22.157.254:38345: OK, instance_id { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" instance_seqno: 1790610993828917 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:56:35.015157 13758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:35.015247 13733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 15:56:35.015357 13839 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:56:35.015456 13733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:35.019995 13733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } }
I20260928 15:56:35.022559 13733 leader_election.cc:288] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [CANDIDATE]: Term 1 election: Requested vote from peers c40f13097b644d8aaca795b95ae73a0d (127.22.157.253:37551), 18582b2d8b4d4e23baa03d7132eb4bd2 (127.22.157.252:38537)
I20260928 15:56:35.023732 13881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3c82b5a0d04f50987723e58ce1684c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2"
I20260928 15:56:35.024242 13881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:35.027963 13881 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e3c82b5a0d04f50987723e58ce1684c in term 1.
I20260928 15:56:35.028959 13708 leader_election.cc:302] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18582b2d8b4d4e23baa03d7132eb4bd2, 2e3c82b5a0d04f50987723e58ce1684c; no voters:
I20260928 15:56:35.029510 13928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 15:56:35.040412 13928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [term 1 LEADER]: Becoming Leader. State: Replica: 2e3c82b5a0d04f50987723e58ce1684c, State: Running, Role: LEADER
I20260928 15:56:35.041853 13928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } }
I20260928 15:56:35.052217 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3c82b5a0d04f50987723e58ce1684c. Latest consensus state: current_term: 1 leader_uuid: "2e3c82b5a0d04f50987723e58ce1684c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:35.053107 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:35.056674 13939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 15:56:35.068307 13712 tablet.cc:2369] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:56:35.073834 13939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 15:56:35.077481 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } attempt: 5
Sep 28 15:56:35 dist-test-slave-1vw0 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790610994, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.157.253@KRBTEST.COM for kudu/127.22.157.252@KRBTEST.COM
I20260928 15:56:35.098165 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:35.078910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.22.157.252:38537)
0928 15:56:35.079077 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.079083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.079101 (+ 18us) server_negotiation.cc:407] Connection header received
0928 15:56:35.079174 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.079179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.079252 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.079570 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.080691 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.081828 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.085264 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.085996 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.086122 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:35.090387 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:35.090394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:35.090399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:35.090445 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:35.094077 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:35.094960 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:35.094965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:35.094969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:35.095123 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:35.095625 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:35.095630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:35.095634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:35.095997 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:35.096173 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:35.097026 (+ 853us) server_negotiation.cc:299] Negotiation successful
0928 15:56:35.097262 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 15:56:35.098450 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:35.078151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.157.252:38537 (local address 127.0.0.1:33670)
0928 15:56:35.078530 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 15:56:35.078549 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 15:56:35.078674 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 15:56:35.079747 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 15:56:35.079753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 15:56:35.080286 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 15:56:35.080502 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:35.080513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.083508 (+ 2995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.083516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:35.085060 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 15:56:35.085078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.086341 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.086348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 15:56:35.086604 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:35.087258 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 15:56:35.087282 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 15:56:35.090154 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 15:56:35.094272 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:35.094278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:35.094282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:35.094805 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:35.095279 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 15:56:35.095284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 15:56:35.095287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 15:56:35.095485 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 15:56:35.096237 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 15:56:35.096242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 15:56:35.096418 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 15:56:35.096574 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 15:56:35.097602 (+ 1028us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260928 15:56:35.104696 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d: Bootstrap starting.
I20260928 15:56:35.085762 13691 master_runner.cc:186] Error getting master registration for 127.22.157.253:37551: OK, instance_id { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" instance_seqno: 1790610994349470 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 15:56:35.107367 13691 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 15:56:35.115271 13832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d: Neither blocks nor log segments found. Creating new log.
I20260928 15:56:35.119388 13832 log.cc:824] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d: Log is configured to *not* fsync() on all Append() calls
I20260928 15:56:35.122103 13881 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [term 1 FOLLOWER]: Refusing update from remote peer 2e3c82b5a0d04f50987723e58ce1684c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:56:35.123580 13936 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [LEADER]: Connected to new peer: Peer: permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 15:56:35.126230 13860 tablet.cc:2369] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 15:56:35.127414 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:35.081942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.22.157.252:38537)
0928 15:56:35.096484 (+ 14542us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.096493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.096517 (+ 24us) server_negotiation.cc:407] Connection header received
0928 15:56:35.096565 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.096571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.096663 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.099632 (+ 2969us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.101852 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.103114 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.107995 (+ 4881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.109172 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.109429 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:35.115816 (+ 6387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:35.115856 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:35.115862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:35.115941 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:35.119991 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:35.120906 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:35.120912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:35.120916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:35.121118 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:35.121583 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:35.121589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:35.121592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:35.122407 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:35.124676 (+ 2269us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:35.124970 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 15:56:35.125457 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14404,"thread_start_us":14760,"threads_started":1}
I20260928 15:56:35.134771 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d: No bootstrap required, opened a new log
I20260928 15:56:35.135581 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:35.072798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.22.157.254:38345)
0928 15:56:35.072995 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.073001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.077624 (+ 4623us) server_negotiation.cc:407] Connection header received
0928 15:56:35.078698 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.078707 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.078815 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.078905 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.103319 (+ 24414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.104413 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.110196 (+ 5783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.110909 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.111018 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:35.124099 (+ 13081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:35.124107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:35.124111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:35.124158 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:35.128425 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:35.129327 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:35.129333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:35.129336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:35.129775 (+ 439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:35.133560 (+ 3785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:35.133566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:35.133570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:35.134029 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:35.134328 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:35.134904 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 15:56:35.135183 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 15:56:35.138743 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:35.072402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.22.157.253:37551)
0928 15:56:35.072634 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.072643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.080713 (+ 8070us) server_negotiation.cc:407] Connection header received
0928 15:56:35.097308 (+ 16595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.097315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.097392 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.097461 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.108016 (+ 10555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.109172 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.113667 (+ 4495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:35.114419 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:35.114525 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:35.129404 (+ 14879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 15:56:35.129414 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 15:56:35.129420 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 15:56:35.129473 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 15:56:35.133323 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:35.134968 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:35.134985 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:35.134989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:35.135171 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 15:56:35.136357 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 15:56:35.136362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 15:56:35.136365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 15:56:35.136769 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 15:56:35.136995 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:35.138095 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0928 15:56:35.138353 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 15:56:35.152292 13832 raft_consensus.cc:348] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } }
I20260928 15:56:35.152823 13832 raft_consensus.cc:729] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40f13097b644d8aaca795b95ae73a0d, State: Initialized, Role: FOLLOWER
I20260928 15:56:35.153412 13832 consensus_queue.cc:260] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } }
I20260928 15:56:35.155777 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3c82b5a0d04f50987723e58ce1684c. Latest consensus state: current_term: 1 leader_uuid: "2e3c82b5a0d04f50987723e58ce1684c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:35.156163 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:35.157337 13950 mvcc.cc:205] Tried to move back new op lower bound from 7334342636003319808 to 7334342635703537664. Current Snapshot: MvccSnapshot[applied={T|T < 7334342636003319808}]
I20260928 15:56:35.157927 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:35.158725 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:35.159132 13832 sys_catalog.cc:564] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 15:56:35.159523 13806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 15:56:35.164422 13806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [term 1 FOLLOWER]: Refusing update from remote peer 2e3c82b5a0d04f50987723e58ce1684c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 15:56:35.167192 13954 mvcc.cc:205] Tried to move back new op lower bound from 7334342636003319808 to 7334342635703537664. Current Snapshot: MvccSnapshot[applied={T|T < 7334342636003319808}]
I20260928 15:56:35.167659 13807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e3c82b5a0d04f50987723e58ce1684c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c40f13097b644d8aaca795b95ae73a0d"
I20260928 15:56:35.173859 13807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e3c82b5a0d04f50987723e58ce1684c in term 1.
I20260928 15:56:35.176110 13928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 15:56:35.192576 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e3c82b5a0d04f50987723e58ce1684c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:35.192972 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:35.198418 13939 catalog_manager.cc:1347] Generated new cluster ID: 953d762624f54a6e88af6ea09d60ff91
I20260928 15:56:35.198554 13939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 15:56:35.199687 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e3c82b5a0d04f50987723e58ce1684c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:35.199949 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:35.230281 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e3c82b5a0d04f50987723e58ce1684c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:35.230937 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:35.231551 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3c82b5a0d04f50987723e58ce1684c. Latest consensus state: current_term: 1 leader_uuid: "2e3c82b5a0d04f50987723e58ce1684c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:35.231922 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 15:56:35.238766 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e3c82b5a0d04f50987723e58ce1684c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c82b5a0d04f50987723e58ce1684c" member_type: VOTER last_known_addr { host: "127.22.157.254" port: 38345 } } peers { permanent_uuid: "c40f13097b644d8aaca795b95ae73a0d" member_type: VOTER last_known_addr { host: "127.22.157.253" port: 37551 } } peers { permanent_uuid: "18582b2d8b4d4e23baa03d7132eb4bd2" member_type: VOTER last_known_addr { host: "127.22.157.252" port: 38537 } } }
I20260928 15:56:35.239081 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c [sys.catalog]: This master's current role is: LEADER
I20260928 15:56:35.250361 13939 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 15:56:35.252741 13939 catalog_manager.cc:1504] Loading token signing keys...
I20260928 15:56:35.278206 13939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e3c82b5a0d04f50987723e58ce1684c: Generated new TSK 0
I20260928 15:56:35.280951 13939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 15:56:35.281673 13974 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 15:56:35.392869 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.388194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.22.157.253:37551)
0928 15:56:35.388378 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.388384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.390206 (+ 1822us) server_negotiation.cc:407] Connection header received
0928 15:56:35.391148 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.391155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.391253 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.391329 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.392608 (+ 1279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55286: BlockingRecv error: recv got EOF from 127.0.0.1:55286 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 15:56:35.398196 13951 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.390812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.22.157.252:38537)
0928 15:56:35.390992 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.390998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.395657 (+ 4659us) server_negotiation.cc:407] Connection header received
0928 15:56:35.395814 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.395820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.395930 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.396040 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.397938 (+ 1898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33686: BlockingRecv error: recv got EOF from 127.0.0.1:33686 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 15:56:35.399258 13912 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.387913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.22.157.254:38345)
0928 15:56:35.388121 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.388128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.394574 (+ 6446us) server_negotiation.cc:407] Connection header received
0928 15:56:35.396067 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.396073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.396168 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.396242 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.398991 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46932: BlockingRecv error: recv got EOF from 127.0.0.1:46932 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 15:56:35.407961 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.405663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.22.157.253:37551)
0928 15:56:35.405857 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.405863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.405993 (+ 130us) server_negotiation.cc:407] Connection header received
0928 15:56:35.406343 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.406350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.406487 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.406585 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.407682 (+ 1097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55292: BlockingRecv error: recv got EOF from 127.0.0.1:55292 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 15:56:35.415904 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.413408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.22.157.253:37551)
0928 15:56:35.413612 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.413618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.414141 (+ 523us) server_negotiation.cc:407] Connection header received
0928 15:56:35.414474 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.414481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.414609 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.414687 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.415680 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55300: BlockingRecv error: recv got EOF from 127.0.0.1:55300 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 15:56:35.431022 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.426380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.22.157.253:37551)
0928 15:56:35.426580 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.426586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.429223 (+ 2637us) server_negotiation.cc:407] Connection header received
0928 15:56:35.429529 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.429549 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.429661 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.429741 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.430767 (+ 1026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55316: BlockingRecv error: recv got EOF from 127.0.0.1:55316 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 15:56:35.442271 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.439605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.22.157.253:37551)
0928 15:56:35.439836 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.439842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.440394 (+ 552us) server_negotiation.cc:407] Connection header received
0928 15:56:35.440762 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.440769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.440885 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.440962 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.442014 (+ 1052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55320: BlockingRecv error: recv got EOF from 127.0.0.1:55320 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260928 15:56:35.457342 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.454898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.22.157.253:37551)
0928 15:56:35.455095 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.455103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.455704 (+ 601us) server_negotiation.cc:407] Connection header received
0928 15:56:35.455931 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.455938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.456067 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.456193 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.457092 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55328: BlockingRecv error: recv got EOF from 127.0.0.1:55328 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 15:56:35.479899 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.477386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.22.157.253:37551)
0928 15:56:35.477609 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.477616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.478176 (+ 560us) server_negotiation.cc:407] Connection header received
0928 15:56:35.478400 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.478408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.478525 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.478649 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.479648 (+ 999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55336: BlockingRecv error: recv got EOF from 127.0.0.1:55336 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 15:56:35.519457 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.517022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.22.157.253:37551)
0928 15:56:35.517234 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.517241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.517783 (+ 542us) server_negotiation.cc:407] Connection header received
0928 15:56:35.518012 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.518019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.518136 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.518259 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.519168 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55338: BlockingRecv error: recv got EOF from 127.0.0.1:55338 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 15:56:35.594310 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.591920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.22.157.253:37551)
0928 15:56:35.592134 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.592141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.592690 (+ 549us) server_negotiation.cc:407] Connection header received
0928 15:56:35.592935 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.592943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.593054 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.593176 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.594070 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55348: BlockingRecv error: recv got EOF from 127.0.0.1:55348 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 15:56:35.731976 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.730004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.22.157.253:37551)
0928 15:56:35.730221 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.730226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.730561 (+ 335us) server_negotiation.cc:407] Connection header received
0928 15:56:35.730753 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.730758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.730838 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.730944 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.731768 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55350: BlockingRecv error: recv got EOF from 127.0.0.1:55350 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 15:56:35.993804 13836 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 15:56:35.991859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.22.157.253:37551)
0928 15:56:35.992111 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:35.992117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:35.992374 (+ 257us) server_negotiation.cc:407] Connection header received
0928 15:56:35.992606 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:35.992611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:35.992717 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:35.992851 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 15:56:35.993610 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55352: BlockingRecv error: recv got EOF from 127.0.0.1:55352 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 15:56:36.008463 13926 catalog_manager.cc:1259] Loaded cluster ID: 953d762624f54a6e88af6ea09d60ff91
I20260928 15:56:36.008565 13926 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2: loading cluster ID for follower catalog manager: success
I20260928 15:56:36.011098 13926 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2: acquiring CA information for follower catalog manager: success
I20260928 15:56:36.013010 13926 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18582b2d8b4d4e23baa03d7132eb4bd2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:56:36.286139 13974 catalog_manager.cc:1259] Loaded cluster ID: 953d762624f54a6e88af6ea09d60ff91
I20260928 15:56:36.286298 13974 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d: loading cluster ID for follower catalog manager: success
I20260928 15:56:36.289013 13974 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d: acquiring CA information for follower catalog manager: success
I20260928 15:56:36.291055 13974 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c40f13097b644d8aaca795b95ae73a0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 15:56:36.517761 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:36.510835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.22.157.253:37551)
0928 15:56:36.511702 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:36.511707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:36.511726 (+ 19us) server_negotiation.cc:407] Connection header received
0928 15:56:36.511801 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:36.511805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:36.511949 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:36.512117 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:36.512629 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.513644 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.515200 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.515686 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.515829 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:36.516451 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:36.517060 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:36.517247 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:36.517303 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 15:56:36.517391 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":490,"threads_started":1}
I20260928 15:56:36.530759 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:36.524563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.22.157.252:38537)
0928 15:56:36.525306 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:36.525313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:36.525335 (+ 22us) server_negotiation.cc:407] Connection header received
0928 15:56:36.525452 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:36.525456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:36.525618 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:36.525799 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:36.526207 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.526939 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.528248 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.528691 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.528861 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:36.529419 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:36.530072 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:36.530246 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:36.530352 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 15:56:36.530442 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":517,"threads_started":1}
I20260928 15:56:36.552697 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:36.537437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.22.157.253:37551)
0928 15:56:36.537705 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:36.537714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:36.544278 (+ 6564us) server_negotiation.cc:407] Connection header received
0928 15:56:36.544946 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:36.544956 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:36.545108 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:36.545192 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:36.546151 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.547739 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.550188 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.550894 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.551002 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:36.551778 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:36.551930 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:36.552018 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:36.552255 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 15:56:36.552344 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 15:56:36.557981 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:36.537126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.22.157.254:38345)
0928 15:56:36.548599 (+ 11473us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:36.548606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:36.550166 (+ 1560us) server_negotiation.cc:407] Connection header received
0928 15:56:36.550506 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:36.550513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:36.550730 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:36.550898 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:36.551740 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.552931 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.555099 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.555744 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.555846 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:36.556687 (+ 841us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:36.557207 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:36.557295 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:36.557539 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 15:56:36.557640 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11121,"thread_start_us":11251,"threads_started":1}
I20260928 15:56:36.560186 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 15:56:36.550495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.22.157.252:38537)
0928 15:56:36.550690 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 15:56:36.550703 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 15:56:36.553752 (+ 3049us) server_negotiation.cc:407] Connection header received
0928 15:56:36.554050 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 15:56:36.554057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 15:56:36.554161 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 15:56:36.554247 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 15:56:36.554799 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.555847 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.557828 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 15:56:36.558424 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 15:56:36.558536 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 15:56:36.559368 (+ 832us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 15:56:36.559516 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 15:56:36.559581 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 15:56:36.559803 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 15:56:36.559893 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 15:56:36.563818 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13691
I20260928 15:56:36.564911 13753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:36.684160 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 13691
W20260928 15:56:36.706209 13855 connection.cc:535] server connection from 127.0.0.1:33656 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 15:56:36.706892 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13765
I20260928 15:56:36.708015 13827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:36.815501 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 13765
I20260928 15:56:36.836529 23159 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task1s6B80/build/asan/bin/kudu pid 13839
I20260928 15:56:36.837761 13901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 15:56:36.945140 23159 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1s6B80/build/asan/bin/kudu with pid 13839
2026-09-28T15:56:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5044 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10110 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (378175 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST